Improved spatio-temporal cross attention transformer system and method for human action recognition

Abstract

The present invention relates to a human behavior recognition system using an improved spatio-temporal cRoss attention transformer, and more specifically, to a behavior recognition system that recognizes human behavior using input video and skeleton data, comprising: a data processing unit that constructs input tokens from the input video and skeleton data; and a behavior recognition unit that inputs the input tokens into an improved spatio-temporal cRoss attention (STAR++) transformer to estimate a final behavior label, wherein the STAR++ transformer is characterized by being composed of a plurality of encoder layers including an interval attention (IAttn) module that performs an attention mechanism, divides the input tokens according to a preset interval term based on the time dimension to create two token groups, and performs an attention mechanism between the two token groups to learn behavioral changes. In addition, the present invention relates to a method for recognizing human behavior using an improved spatio-temporal cRoss attention transformer, and more specifically, to a method for recognizing human behavior using input video and skeleton data, wherein each step is performed on a computer, comprising: (1) a step of constructing an input token from the input video and skeleton data; and (2) a step of inputting the input token into an improved spatio-temporal cRoss attention (STAR++) transformer to estimate a final behavior label, wherein the STAR++ transformer is characterized by being composed of a plurality of encoder layers including an interval attention (IAttn) module that performs an attention mechanism, divides the input token according to a preset interval term based on the time dimension to create two token groups, and performs an attention mechanism between the two token groups to learn behavioral changes. According to the human behavior recognition system and method using an improved spatiotemporal cross-attention transformer proposed in the present invention, behavior recognition is performed by learning video frames and human skeletal features together; moreover, data from heterogeneous domains can be effectively learned by flexibly combining image and skeletal-based data through multi-representation learning and applying multi-class tokens, and behavior recognition performance can be enhanced by learning spatiotemporal features together through a transformer (STAR++ transformer) with improved spatiotemporal cross-attention applied. In addition, according to the human behavior recognition system and method using the improved spatiotemporal cross-attention transformer proposed in the present invention, the STAR++ transformer is configured with a plurality of encoder layers including an interval attention module that divides two token groups according to a preset interval term based on the time dimension and learns behavioral changes by performing a cross-attention mechanism between the two token groups, thereby enabling progressive learning of the difference between large and small features on the time axis according to depth, and simplifies the network by integrating it into a single encoder form rather than an encoder-decoder structure, thereby reducing computational costs.

Description

Technology Field

[0001] The present invention relates to a human behavior recognition system and method, and more specifically, to a human behavior recognition system and method using an improved spatiotemporal cross-attention transformer. Background Technology

[0002] The content described in this section merely provides background information regarding an embodiment of the present invention and does not constitute prior art.

[0004] In recent deep learning research related to computer vision, multi-modal learning, which models combinations of various data formats, is gaining attention. Among the many application fields related to computer vision, video motion recognition is an example where performance can be improved using a multi-modal approach. While existing video motion recognition methods generally use features such as video sequences or skeletal features independently, multi-modal approaches that also utilize audio, optical flow, motion, and texture have recently been studied. Existing methods for video motion recognition using multi-modal learning tend to follow two trends: modeling RGB frames and optical flow. However, because these methods derive results using multi-modal data separately and then combine them, expressing prediction results for the same information is time-consuming and ambiguous.

[0006] Vision Transformer (ViT)-based models for time-series processing have demonstrated excellent performance through multi-head self-attention mechanisms, but they require significant computational cost. To address this computational issue, efficient Transformer-based video recognition algorithms have recently been researched. Furthermore, there is an increasing number of Transformer-based models that apply deformable token selection methods or methods to progressively reduce or delete tokens. These models maintain or improve performance by using deformable token selection and token halting techniques to reduce token size, thereby effectively decreasing self-attention operations, which account for the majority of computational cost.

[0008] Meanwhile, as prior art related to the present invention, published patent No. 10-2022-0078893 (Title of invention: Device and method for recognizing the behavior of a person in an image, Date of publication: June 13, 2022) and others have been disclosed.

[0010] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as publicly known technology disclosed to the general public prior to the filing of the present invention. The problem to be solved

[0011] The present invention is proposed to solve the aforementioned problems of previously proposed methods, and aims to provide a human behavior recognition system and method using an improved spatiotemporal cross-attention transformer that performs behavior recognition by learning video frames and human skeletal features together, flexibly combines image and skeletal-based data through multi-representation learning, effectively learns heterogeneous domain data by applying multi-class tokens, and enhances behavior recognition performance by learning spatiotemporal features together through a transformer (STAR++ transformer) with improved spatiotemporal cross-attention.

[0013] In addition, another objective of the present invention is to provide a human behavior recognition system and method using an improved spatiotemporal cross-attention transformer, wherein the STAR++ transformer is configured with a plurality of encoder layers including an interval attention module that divides two token groups according to a preset interval term based on the time dimension and learns behavioral changes by performing a cross-attention mechanism between the two token groups, thereby enabling progressive learning of the difference between large and small features on the time axis according to depth, and simplifying the network by integrating it into a single encoder form rather than an encoder-decoder structure to reduce computational costs.

[0015] However, the technical problem that the present invention aims to solve is not limited to the technical problem described above, and other technical problems may exist. It goes without saying that objectives or effects that can be understood from the means of solving the problem or the embodiments, even if not explicitly mentioned, are also included. means of solving the problem

[0016] A human behavior recognition system using an improved spatiotemporal intersection attention transformer according to the features of the present invention for achieving the above-mentioned purpose,

[0017] As a behavior recognition system that recognizes human behavior using input video and skeleton data,

[0018] A data processing unit that constructs input tokens from the above input video and skeletal data; and

[0019] It is configured to include a behavior recognition unit that inputs the above input token into an improved Spatio-TemproAl cRoss attention (STAR++) transformer to estimate the final behavior label, and

[0020] The above STAR++ Transformers are,

[0021] The configuration is characterized by being composed of multiple encoder layers, including an Interval Attention (IAttn) module that performs an attention mechanism, divides the input tokens according to a preset interval term based on the time dimension to create two token groups, and learns behavioral changes by performing a cross-attention mechanism between the two token groups.

[0023] Preferably, the above interval term is,

[0024] Each of the above encoder layers is preset, but the value can be set to increase as the layer deepens.

[0026] Preferably, the data processing unit is,

[0027] Global features and local features are extracted from the above input video frame, and an input token can be constructed by combining a global grid token obtained by tokenizing the global features and a joint map token obtained by combining the local features and skeletal data, including a multi-class token.

[0029] More preferably, the data processing unit is,

[0030] A joint heatmap generated using the joint coordinates of the above skeletal data can be tokenized by combining it with the above local features to generate the above joint map token.

[0032] More preferably, the data processing unit is,

[0033] The input token can be constructed by combining the global grid token and the joint map token, and connecting the multi-class token between both ends of the entire token and the global grid token and the joint map token.

[0035] Preferably, the encoder layer is,

[0036] It may be configured to include a full attention module that performs a full spatio-temporal attention (FAttn) mechanism for all tokens constituting the input tokens; and an interval attention module that divides the output of the full attention module according to a preset interval term based on the time dimension to generate two token groups and performs an intersection attention mechanism between the two token groups.

[0038] More preferably, the STAR++ transformer is,

[0039] It may further include a D3D (Deformable 3D) token selection module that extracts the global grid token from the input token, combines the offset and reference point to generate final coordinates, and generates a key and a value from the final coordinates.

[0041] Even more preferably, the encoder layer is,

[0042] (a) A step in which the D3D token selection module generates a key and a value from the input token;

[0043] (b) A step in which the above-mentioned total attention module performs a total spatiotemporal attention mechanism using the key and value generated in the above-mentioned D3D token selection module;

[0044] (c) A step of generating two token groups by dividing the output of the above-mentioned entire attention module according to a preset interval term based on the time dimension;

[0045] (d) a step of generating keys and values ​​from two token groups in which the above D3D token selection module is generated; and

[0046] (e) The above interval attention module may perform a step of performing an intersection attention mechanism between two token groups generated in step (c) using the key and value generated in step (d).

[0048] A human behavior recognition method using an improved spatiotemporal intersection attention transformer according to the features of the present invention for achieving the above-mentioned purpose,

[0049] As a behavior recognition method that recognizes human behavior using input video and skeleton data, with each step performed on a computer,

[0050] (1) A step of constructing input tokens from the above input video and skeletal data; and

[0051] (2) The above input token is input into an improved Spatio-TemproAl cRoss attention (STAR++) transformer to estimate the final action label, and

[0052] The above STAR++ Transformers are,

[0053] The configuration is characterized by being composed of multiple encoder layers, including an Interval Attention (IAttn) module that performs an attention mechanism, divides the input tokens according to a preset interval term based on the time dimension to create two token groups, and learns behavioral changes by performing a cross-attention mechanism between the two token groups.

[0055] Preferably, the above interval term is,

[0056] Each of the above encoder layers is preset, but the value can be set to increase as the layer deepens.

[0058] Preferably, in step (1),

[0059] Global features and local features are extracted from the above input video frame, and an input token can be constructed by combining a global grid token obtained by tokenizing the global features and a joint map token obtained by combining the local features and skeletal data, including a multi-class token.

[0061] More preferably, in step (1),

[0062] A joint heatmap generated using the joint coordinates of the above skeletal data can be tokenized by combining it with the above local features to generate the above joint map token.

[0064] More preferably, in step (1),

[0065] The input token can be constructed by combining the global grid token and the joint map token, and connecting the multi-class token between both ends of the entire token and the global grid token and the joint map token.

[0067] Preferably, the encoder layer is,

[0068] It may be configured to include a full attention module that performs a full spatio-temporal attention (FAttn) mechanism for all tokens constituting the input tokens; and an interval attention module that divides the output of the full attention module according to a preset interval term based on the time dimension to generate two token groups and performs an intersection attention mechanism between the two token groups.

[0070] More preferably, the STAR++ transformer is,

[0071] It may further include a D3D token selection module that extracts the global grid token from the input token, combines the offset and reference point to generate final coordinates, and generates a key and a value from the final coordinates.

[0073] Even more preferably, the encoder layer is,

[0074] (a) A step in which the D3D token selection module generates a key and a value from the input token;

[0075] (b) A step in which the above-mentioned total attention module performs a total spatiotemporal attention mechanism using the key and value generated in the above-mentioned D3D token selection module;

[0076] (c) A step of generating two token groups by dividing the output of the above-mentioned entire attention module according to a preset interval term based on the time dimension;

[0077] (d) a step of generating keys and values ​​from two token groups in which the above D3D token selection module is generated; and

[0078] (e) The above interval attention module may perform a step of performing an intersection attention mechanism between two token groups generated in step (c) using the key and value generated in step (d). Effects of the invention

[0079] According to the human behavior recognition system and method using an improved spatiotemporal cross-attention transformer proposed in the present invention, behavior recognition is performed by learning video frames and human skeletal features together; moreover, data from heterogeneous domains can be effectively learned by flexibly combining image and skeletal-based data through multi-representation learning and applying multi-class tokens, and behavior recognition performance can be enhanced by learning spatiotemporal features together through a transformer (STAR++ transformer) with improved spatiotemporal cross-attention applied.

[0081] In addition, according to the human behavior recognition system and method using the improved spatiotemporal cross-attention transformer proposed in the present invention, the STAR++ transformer is configured with a plurality of encoder layers including an interval attention module that divides two token groups according to a preset interval term based on the time dimension and learns behavioral changes by performing a cross-attention mechanism between the two token groups, thereby enabling progressive learning of the difference between large and small features on the time axis according to depth, and simplifies the network by integrating it into a single encoder form rather than an encoder-decoder structure, thereby reducing computational costs.

[0083] Furthermore, the various and beneficial advantages and effects of the present invention are not limited to those described above and may be more easily understood in the process of explaining specific embodiments of the present invention. Brief explanation of the drawing

[0084] Figure 1 is a diagram showing the spacetime intersection of a conventional STAR-Transformer. FIG. 2 is a diagram showing the spatiotemporal intersection attention of a STAR++ transformer proposed in a human behavior recognition system using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention. FIG. 3 is a diagram illustrating the configuration of a human behavior recognition system using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention. FIG. 4 is a drawing illustrating the overall structure of a human behavior recognition system using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the detailed structure of the encoder layer of a STAR++ transformer in a human behavior recognition system using an improved spacetime intersection attention transformer according to an embodiment of the present invention. FIG. 6 is a diagram illustrating the D3D token selection structure of a D3D token selection module in a human behavior recognition system using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention. FIG. 7 is a diagram illustrating Interval Attention (IAtn) in a human behavior recognition system using an improved spatiotemporal cross-attention transformer according to an embodiment of the present invention. FIG. 8 is a diagram illustrating the pseudo code of the IAttn process performed in the interval attention module in a human behavior recognition system using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention. FIG. 9 is a diagram illustrating the flow of a human behavior recognition method using a space-time intersection attention transformer according to an embodiment of the present invention. FIG. 10 is a diagram illustrating the visualization results of NTU-RGB+D 60 and Penn behavior datasets using a human behavior recognition system and method using a spatiotemporal cross-attention transformer according to an embodiment of the present invention. FIG. 11 is a diagram illustrating the visualization of attention scores for joint map tokens representing skeletal information in a human behavior recognition system and method using a spatiotemporal intersection attention transformer according to an embodiment of the present invention. Specific details for implementing the invention

[0085] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0087] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected" but also cases where they are "indirectly connected" with other elements interposed between them. Furthermore, when a part is described as "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for the inclusion of additional components; it should be understood that this does not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0089] The following examples are detailed descriptions to aid in understanding the present invention and are not intended to limit the scope of the present invention. Accordingly, inventions within the same scope that perform the same function as the present invention will also fall within the scope of the present invention.

[0091] In addition, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to one another.

[0093] In addition, some of the operations or functions described as being performed by a terminal, device, or device in the present invention may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.

[0095] In particular, the means for executing the system according to each embodiment of the present invention may be an application or a web server, and the terminal, which is the means for reading the recording medium on which the application or web server is recorded, may include not only general PCs such as general desktops or laptops, but also mobile terminals such as smartphones and tablet PCs.

[0097] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0099] The present invention relates to a human behavior recognition system and method using an improved spatiotemporal cross-attention transformer. The human behavior recognition system and method using an improved spatiotemporal cross-attention transformer according to the features of the present invention may be composed of software recorded on hardware including memory and a processor. For example, the human behavior recognition system and method using an improved spatiotemporal cross-attention transformer may be stored and implemented on a personal computer, laptop computer, server computer, PDA, smartphone, tablet PC, etc. For convenience of explanation, the entity performing each step may be omitted below.

[0101] FIG. 1 is a diagram showing the spatio-temporal attention of a conventional STAR-Transformer, and FIG. 2 is a diagram showing the spatio-temporal attention of an improved spatio-temporal attention (improved Spatio-TemproAl cRoss attention, STAR++) transformer proposed in a human behavior recognition system using an improved spatio-temporal attention transformer according to an embodiment of the present invention.

[0103] In particular, the STAR-Transformer (D. Ahn, S. Kim, H. Hong and BC Ko, “STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition,” in Proc. WACV, Waikoloa, HI, USA, 2023, pp. 3330-3339.), used in multi-modal action recognition models, proposes a spatio-temporal cross-attention mechanism that can flexibly combine multiple features into a single input vector, efficiently reduce the number of computations, and learn temporal differences in actions, unlike the 2-stream method, an existing approach that uses a multi-modal learning mechanism with multiple class tokens. However, the encoder-decoder structure including spatio-temporal cross-attention in the STAR-Transformer is still inefficient and requires improvement. Therefore, to further increase the efficiency and improve the performance of the STAR-Transformer, we propose an improved STAR++ structure that includes deformable 3D (D3D) token selection.

[0105] As illustrated in FIG. 1, the STAR-Transformer consists of a zigzag encoder that divides the entire input sequence into a zigzag pattern and a binary decoder that divides it again into half frames. On the other hand, as illustrated in FIG. 2, the STAR++ Transformer can be composed of a single interval attention encoder that uses an interval term to adjust the number of decoupling frames per layer depth. Here, the interval term can generate attention by selecting only the necessary tokens as key and value.

[0107] More specifically, the proposed STAR++ applies a spatiotemporal cross-attention mechanism by separating all tokens into two sets of tokens centered on the time axis, similar to the basic STAR-Transformer method. However, as illustrated in Fig. 1, the STAR-Transformer has an encoder and decoder structure, requiring more parameters and computations as the layer depth increases. In contrast, as illustrated in Fig. 2, STAR++ consists of a single encoder that adjusts the number of decoupling frames per layer depth using an interval term, and significantly reduces the number of computations by applying attention by selecting only the necessary tokens as key and value. Furthermore, STAR++ can progressively learn global feature differences from local feature differences along the time axis. This is made possible through a new Interval Attent (IAtn) that creates tokens using spatiotemporal cross-attention.

[0109] The contribution of the present invention is as follows:

[0110] We propose a new STAR++ motion recognition algorithm that can flexibly combine multi-modal learning by improving the cross-modal learning mechanism of the basic STAR-Transformer.

[0111] - The proposed STAR++ improves upon the encoder-decoder structure of the basic STAR-transformer by using only the encoder structure.

[0112] - We propose an IAttn based on spatiotemporal cross-attention to improve recognition performance by progressively learning local and global features.

[0113] STAR++ proposes D3D token selection, enabling dynamic token selection that efficiently learns the coordinates of tokens that influence the learning results.

[0115] Below, we first describe the basic model, the STAR-Transformer. For tokenization, global and local features can be extracted from input video frames using ResNet Mixed Convolution 18 (MC18) (D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri, "A closer look at spatiotemporal convolutions for action recognition," in Proc. CVPR, Salt Lake City, UT, USA, 2018, pp. 6450-6459.). Since global features contain global information of the video frame, global grid tokens representing RGB features are used. It is generated as follows. Here, T represents the number of video frames, P represents the number of global grid tokens tokenized by partitioning global features into non-overlapping sizes, and C represents the dimension. Since local features contain local information, they are combined with a joint heatmap generated using skeleton coordinates and tokenized. is generated. Here, N is the number of joints. The formula for creating the joint map token is the following mathematical formula 1.

[0117]

[0119] Here, represents a local feature map, and is a joint heat map created from skeletal coordinates.

[0121] The two tokenized multi-modal tokens are combined with a multi-class token and used as input to the STAR-Transformer. The multi-class token is connected between the two ends of the entire token and the multi-modal token, along with the global grid token (T) and the position embedding (pos). g ) and joint map tokens (T j ) can be learned evenly.

[0123]

[0125] Here, C g is T g It is a class token for learning, and C j is T j It is a class token for learning, and C tot is a class token for learning all multi-modal tokens. The final token Z is used as the input to the STAR-Transformer.

[0127] To learn spatio-temporal features, the STAR-Transformer can use full spatio-temporal attention (FAttn), spatio-temporal cross-attention (STCA), zigzag attention (ZAttn), and binary attention (BAttn). FAttn can learn the relationships between all tokens and self-attention by utilizing all tokens in the spatio-temporal dimension. However, FAttn alone is not sufficient to learn the time differences required for motion recognition. Therefore, the STAR-Transformer uses ZAttn and BAttn to sequentially learn the fine and coarse differences of video sequences.

[0129] ZAttn and BAttn first undergo a process of separating all tokens into time-axis-centered frames. As illustrated in Fig. 1, ZAttn separates them in a zigzag pattern by skipping frames centered on the time axis T. BAttn divides the frame in half centered on the time axis T to decouple tokens corresponding to the previous frame from tokens corresponding to the next frame, demonstrating binary token decoupling. At this time, ZAttn and BAttn respectively Q, K, V ∈ R according to the order in which all tokens are separated. S×(T / 2) and Q′,K′,V′∈R S×(T / 2) Two sets of tokens can be generated. Subsequently, these two sets of tokens, O and O′, are used to concatenate the outputs of cross-attention to generate the final STCA result. The formula for this can be expressed as shown in the following mathematical equations 3 to 5.

[0131]

[0133]

[0135]

[0137] By performing a spatiotemporal cross-attention mechanism using this STCA, the STAR-Transformer can learn the feature differences between large video frames and small video frames.

[0139] FIG. 3 is a diagram illustrating the configuration of a human behavior recognition system (10) using an improved spatiotemporal cross-attention transformer according to an embodiment of the present invention, and FIG. 4 is a diagram illustrating the overall structure of a human behavior recognition system (10) using an improved spatiotemporal cross-attention transformer according to an embodiment of the present invention. As shown in FIG. 3 and FIG. 4, the human behavior recognition system (10) using an improved spatiotemporal cross-attention transformer according to an embodiment of the present invention is a behavior recognition system that recognizes human behavior using input video and skeleton data, and may be configured to include a data processing unit (100) and a behavior recognition unit (200). Here, the STAR++ transformer (20) may be composed of multiple encoder layers including an interval attention (IAttn) module that performs an attention mechanism, divides input tokens according to a preset interval term based on the time dimension to create two token groups, and performs an inter-attention mechanism between the two token groups to learn behavioral changes.

[0141] The tokenization process using multi-modal features illustrated in FIG. 4(a) is performed in the data processing unit (100), which is identical to the tokenization process of the STAR-Transformer. As illustrated in FIG. 4(b), the STAR++ Transformer (20) may include D3D token selection and an IAttn process based on the interval spatio-temporal cross attention.

[0143] The data processing unit (100) can construct input tokens from input video and skeletal data. More specifically, the data processing unit (100) can extract global features and local features from an input video frame, and construct an input token by combining a global grid token, which tokenizes the global features, with a joint map token, which tokenizes the local features and skeletal data, including a multi-class token. At this time, the data processing unit (100) can generate a joint map token by tokenizing a joint column map created using the joint coordinates of the skeletal data and combining it with local features. Then, the data processing unit (100) can construct an input token by combining the global grid token and the joint map token, and connecting a multi-class token between the two ends of the entire token and the global grid token and the joint map token.

[0145] That is, in the STAR++ Transformer (20), input tokens can be configured in the same way as the aforementioned STAR-Transformer. More specifically, a video frame and a skeleton can be received as input, and global and local features can be extracted from the video frame through MC18. Global features can be tokenized into global grid tokens representing RGB, and local features can be tokenized into joint map tokens representing the skeleton. Then, the two multi-modal tokens can be flexibly processed by combining them with multi-class tokens and used as input for STAR++.

[0147] The behavior recognition unit (200) can input input tokens into the STAR++ transformer (20) to estimate the final behavior label. More specifically, in the behavior recognition unit (200), STAR++ can learn the differences in video features over time using D3D token selection and IAttn. Finally, after averaging the learned multi-class tokens, a simple multi-layer perceptron (MLP) can be used to derive probability values ​​for the final video recognition. As illustrated in FIGS. 2 and 4, the STAR++ transformer (20) may be composed of multiple encoder layers including an Interval Attention (IATtn) module that performs an attention mechanism, divides input tokens according to a preset interval term based on the time dimension to create two token groups, and performs a cross-attention mechanism between the two token groups to learn behavioral changes. Here, the interval term is preset for each encoder layer, but can be set so that the value increases as the layer deepens.

[0149] FIG. 5 is a diagram illustrating the detailed structure of the encoder layer of a STAR++ transformer (20) in a human behavior recognition system (10) using an improved spatio-temporal attention transformer according to an embodiment of the present invention. As shown in FIG. 4 and 5, in the human behavior recognition system (10) using an improved spatio-temporal attention transformer according to an embodiment of the present invention, the STAR++ transformer (20) is composed only of an encoder layer, unlike the STAR-Transformer which has an encoder-decoder structure, and may be composed of an L-layer encoder layer. The encoder layer comprises a full attention module (220) that performs a Full Spatio-temporal Attention (FAttn) mechanism for all tokens constituting the input token; The total attention module (220) may be configured to include an interval attention module (230) that divides the output of the total attention module (220) according to a preset interval term based on the time dimension to create two token groups and performs an inter-attention mechanism between the two token groups, and may further include a D3D token selection module (210). More specifically, each encoder layer may be computed in the order of the D3D token selection module (210), the total attention module (220), the D3D token selection module (210), and the interval attention module (230).

[0151] Hereinafter, with reference to FIGS. 3 to 5, the detailed configuration of the STAR++ transformer (20) in the human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention will be described in detail.

[0153] The D3D token selection module (210) can process the selection of deformable 3D tokens (D3D tokens). That is, the D3D token selection module (210) can extract global grid tokens from input tokens, combine offsets and reference points to generate final coordinates, and generate keys and values ​​from the final coordinates. D3D token selection is a dynamic token selection method that can capture deformable spatiotemporal features of 3D RGB.

[0155] The self-attention mechanism of Transformers is specialized for determining the spatial relationships between images, demonstrating good performance in image classification and object detection, but it requires significant computational cost. Consequently, there is a growing number of Transformer-based models that apply deformable token selection methods or methods to progressively reduce or delete tokens. These models maintain or improve performance by utilizing deformable token selection and token truncation techniques to reduce token size, effectively decreasing the self-attention operations that account for the majority of computational cost.

[0157] The D3D token selection module (210) can apply a new dynamic token selection method inspired by the approach of techniques that use offsets to learn the location of tokens that influence the learning result and apply token selection (Z. Xia, X. Pan, S. Song, LE Li and G. Huang, "Vision transformer with deformable attention," in Proc. CVPR, New Orleans, LA, USA, 2022, pp. 4794-4803. and M. Jaderberg, K. Simonyan and A. Zisserman, "Spatial transformer networks," in Proc. NeurIPS, Montreal, QC, Canada, 2015, pp. 1-9.). Since this D3D token selection module (210) dynamically selects tokens, it can improve performance by selecting and learning tokens that have a significant impact on the result. Additionally, since the token size can be reduced, operating costs can be effectively reduced.

[0159] FIG. 6 is a diagram illustrating the D3D token selection structure of a D3D token selection module (210) in a human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention. As shown in FIG. 6, the D3D token selection module (210) of the human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention can extract only global grid token Xg from a token set X, combine an offset and a reference point, and generate usable key and value tokens through a token selection process. Below, the 3D3 token selection will be described in detail with reference to FIG. 6.

[0161] Token set X can be configured as follows.

[0163]

[0165] Here, Xg cls is the class token for the global grid, Xg is the global grid token, Xj cls is the class token for the joint map, Xj is the joint map token, Xtot cls is a multi-class token. Here, || represents a link.

[0167] As illustrated in Fig. 6, to generate the final key (K) and value (V), only the global grid token Xg corresponding to the RGB image from the token set X can be extracted and fed into the D3D token selection. Then, an offset θ for learning the coordinates is used with 3D convolution. offset You can generate the final coordinates by creating a function and adding a reference point p.

[0169]

[0171] Here, φ is a function that learns and samples the coordinate values ​​of tokens using a reference point and an offset. Next, W, which are the learnable key and value weights. k and W v Multiply by the temporal key and value It can generate.

[0173]

[0175] and Since the coordinates are real numbers, the coordinates of the 3D token can be calculated through an integer process using two-linear interpolation.

[0177] As shown in Fig. 6, temporal and is Xg cls , Xj cls , Xj, Xtot cls The final K and V can be generated by connecting with.

[0179]

[0181]

[0183] D3D token selection can be performed prior to FAttn and IAttn, respectively, as shown in Fig. 5. First, FAttn is applied using K and V generated through D3D token selection, and token decoupling can be performed through layer normalization. Then, a second D3D token selection is applied, and after applying IAttn, an output vector can be generated through layer normalization and MLP.

[0185] The total attention module (220) can perform a total spacetime attention mechanism on all tokens constituting the input tokens. Existing Transformer models that learn spacetime data generally use total spacetime attention, which performs an attention mechanism on all tokens of the spacetime dimension T. Since the attention mechanism is applied to all tokens of the time dimension T and the space dimension S, the complexity is O(T 2 S 2 It becomes very large with ).

[0187] The interval attention module (230) can learn behavioral changes by dividing the input tokens according to a preset interval term based on the time dimension to create two token groups and performing a cross-attention mechanism between the two token groups.

[0189] Conventional STAR-Transformers demonstrate excellent motion recognition performance, but because they are separated into encoders and decoders and use twice the number of layers, they are not optimized for feature learning based on layer depth. In this invention, instead of using ZAttn and BAttn of the STAR-Transformer, IAttn is proposed, which can progressively learn the difference between large and small features along the time axis according to layer depth. To reduce the computational complexity of the STAR-Transformer, IAttn can integrate the encoder-decoder structure into a single encoder. Therefore, unlike the STAR-Transformer which uses (L×2) layers, STAR++ can use only L layers, thus simplifying the network.

[0191] In the layered structure of STAR++ proposed in the present invention, as shown in FIG. 5, relationships for all tokens can be learned through FAtn (Full Attention), and after normalization (Norm), IAttn token decoupling can be applied. Here, the interval term is pre-set for each encoder layer, but can be set so that its value increases as the layer deepens. That is, the value of the decoupling interval term can gradually increase according to the depth of the layer. For example, a structure with interval terms of I={1,4,8} or I={1,2,4} can be used for three layers (L=3).

[0193] FIG. 7 is a diagram illustrating interval attention (IAtn) in a human behavior recognition system (10) using an improved spatiotemporal cross-attention transformer according to an embodiment of the present invention. FIG. 7 illustrates an example using interval term I={1,2,4}.

[0195] Since the interval term in the l-th layer is 1, the token set is separated at one-frame intervals, similar to the ZAttn of the STAR-Transformer (green token). The decoupling token is generated from a total of two sets of Q, K, and V tokens, and the K and V token sets can generate K and V to be used in IAttn through D3D token selection (blue token). After applying cross-attention using Q, K, and V, the two outputs are concatenated and can be used as inputs for the (l+1) layer.

[0197] In the (l+1)th layer, the token set can be separated so that the interval term becomes 2 and there are two frame intervals. D3D token selection and IAttn can be applied in the same way as in the lth layer. In the Lth layer, the interval term is set to 4 and the same D3D token selection and IAttn can be applied.

[0199] FIG. 8 is a diagram illustrating the pseudo code of an IAttn process performed in an interval attention module (230) in a human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention.

[0201] As illustrated in FIGS. 3 to 5, unlike the basic STAR-Transformer, STAR++ consists only of L encoder layers rather than an encoder-decoder structure. Although the encoder-decoder structure of the basic STAR-Transformer consists of L layers, it actually uses (L×2) layers because it applies cross-attention including ZAttn and BAttn. However, since STAR++ has been improved to use only encoder layers and L layers, it is more efficient in terms of computational complexity than the basic model. In addition, it can learn incrementally by focusing on the supply of global features as the layers deepen. That is, it can be said that STAR++ learns by taking into account the characteristic of the Transformer that focuses on global features as the layers deepen.

[0203] As illustrated in FIG. 5, the encoder layer of the STAR++ transformer (20) can perform the following steps: a step (S10) in which a D3D token selection module (210) generates a key and a value from an input token; a step (S20) in which a total attention module (220) performs a total spacetime attention mechanism using the key and value generated by the D3D token selection module (210); a step (S30) in which the output of the total attention module (220) is divided according to a preset interval term based on the time dimension to generate two token groups; a step (S40) in which the D3D token selection module (210) generates a key and a value from the two generated token groups; and a step (S50) in which an interval attention module (230) performs a cross-attention mechanism between the two token groups generated in step S30 using the key and value generated in step S40.

[0205] The structure of the STAR++ encoder layer illustrated in Fig. 5 can be expressed through the following mathematical equations 11 to 13.

[0207]

[0209]

[0211]

[0213] Here, l is the number of transformer layers, and LN is the layer normalization. Finally, the behavior recognition unit (200) can average the multi-class tokens learned based on the formula and predict behavior labels using a simple MLP.

[0215] FIG. 9 is a diagram illustrating the flow of a human behavior recognition method using a spatiotemporal cross-attention transformer according to an embodiment of the present invention. As shown in FIG. 9, the human behavior recognition method using a spatiotemporal cross-attention transformer according to an embodiment of the present invention is a behavior recognition method that recognizes human behavior using video and skeleton data in which each step is performed on a computer, and can be implemented by including a step (S100) of constructing an input token from input video and skeleton data and a step (S200) of inputting the input token into a STAR++ transformer (20) to estimate a final behavior label.

[0217] Since the details regarding each step have been sufficiently explained in relation to the human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer according to an embodiment of the present invention, a detailed explanation will be omitted.

[0219] experiment

[0220] Details regarding the proposed STAR++, including the training dataset and hyperparameters, can be described. Additionally, performance comparisons and quantitative evaluations were performed using state-of-the-art (SoTA) models on the corresponding benchmark dataset. A dissection study was conducted to demonstrate the effectiveness of each module, including D3D token selection and IAttn, of the proposed STAR++.

[0222] A. Data set

[0223] Extensive experiments were conducted using the action recognition benchmark datasets, Penn-Action dataset, NTU-RGB+D 60, and NTU-RGB+D 120.

[0225] The Penn-Action dataset contains 2,326 video sequences of 15 sports actions, including baseball_pitch, clean_and_jerk, and pull_ups, and 13 human skeletal annotations for each video sequence.

[0227] NTU-RGB+D 60 (hereinafter NTU60) is a large dataset containing 60 behavioral classes and 56,880 video samples for daily, health-related, and interactional behaviors. The dataset provides RGB video sequences, depth map sequences, 3D skeletal data, and infrared (IR) video clips for each sample. The pixel resolution of the RGB video is 1920×1080, and the pixel resolution of both the depth map sequences and IR video is 512×512. The 3D skeletal data contains 25 joint coordinates for each frame. Evaluation using NTU60 followed two methods: cross-subject (XSub) and cross-view (XView). (1) For XSub, data from 20 of the 40 subjects' behaviors were used for training, and data from the remaining 20 subjects were used for evaluation. (2) For XView, of the three camera views, two were used for training and one was applied for evaluation.

[0229] NTU-RGB+D 120 (hereinafter NTU120) is an extension of NTU60 that adds 60 action classes and 57,600 video samples, and includes a total of 120 action classes and 114,480 video samples. Evaluation using NTU120 was performed using two methods: XSub and Xset. (1) For XSub, data from 53 of the 106 subjects' actions were used for training, and data from the remaining 53 subjects were used for evaluation. (2) For Xset, all samples had an even set ID for training and an odd set ID for testing. That is, 16 sets were used for training, and the remaining 16 sets were used for evaluation.

[0231] B. Implementation Details

[0233] The proposed STAR++ Transformer (20) was implemented using PyTorch and ResNet MC18 pre-trained with Kinetics-400 (W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman and A. Zisserman, “The kinetics human action video dataset,” 2017, arXiv preprint arXiv:1705.06950.) as the backbone. A total of 16 frames were used to train the STAR++ Transformer (20), with a batch size of 8, 100 training epochs, a stochastic gradient descent optimizer, a training rate of 2e-4, and a momentum of 0.9. Experiments were performed in an environment consisting of four NVIDIA Tesla V100 GPUs.

[0235] C. Comparison with state-of-the-art methods

[0237] We compared the performance of the SoTA method and STAR++ using Penn-Action and the motion recognition benchmark datasets NTU-RGB+D 60 and 120.

[0239] NTU-RGB+D 60 and 120 results

[0240] 1-inch NTU-RGB+D 60 × 120 x 120 x 120 x 120 x 120 x. In Proc. CVPR, Salt Lake City, UT, USA, 2018, pp. 1159-1168.), MMTM (V. Reza, H. Joze, A. Shaban, M. L. Yuan, pp. 1159-1168). Iuzzolino and K. Koishida, "Mmtm: Multimodal transfer module for cnn fusion," in Proc Proc. ECCV, Virtual, 2020, pp. 72-90.), DualHead-Net (T. Chen, D. Zhou, J. Wang, S. Wang, Y. Guan, X. He and E. Ding, "Learning multi-granular spatio-temporal graph network for skeleton- based action recognition," in Proc. China, 2021, pp. 4334-4342.), Skeletal GNN (A. Zeng, X. Sun, L. Yang, N. Zhao, M. Liu and Q. Xu, "Learning skeletal graph neural networks for hard 3d pose estimation," in Proc.), CTR-GCN (Y. Chen, Z. Zhang, C. Yuan, B. Li, Y. Deng and W. Hu, “Channel-wise topology refinement graph convolution for skeleton-based action recognition,” in Proc. ICCV, Virtual, 2021, pp. 13359-13368.), InfoGCN (H. G. Chi, M. H. Ha, S. Chi, S. W. Lee, Q. Huang and K. Ramani, "Infogcn: Representation learning for human skeleton-based action recognition," in Proc. CVPR, New Orleans, LA, USA, 2022, pp. 20186- 20196.), 3s-AimCLR (T. Guo, H. Liu, Z. Chen, M. Liu, T. Wang and R. Ding, "Contrastive learning from extremely augmented skeleton sequences for self- supervised action recognition," in Proc. AAAI, Washington, DC, USA, 2022, pp. 762-770.), PoseC3D (H. Duan, Y. Zhao, K. Chen, D. Lin and B. Dai, "Revisiting skeleton- based action recognition," in Proc. CVPR, New Orleans, LA, USA, 2022, pp. 2969-2978.), KA-AGTN (Y. Liu, H. Zhang, D. Xu and K. He, "Graph transformer network with temporal kernel attention for skeleton-based action recognition," Knowledge-Based Syst., vol. 240, pp.108146, Jan. 2022, doi: 10.1016 / j.knosys.2022.108146.), compared with STAR (STAR ​​Transformer).

[0242]

[0244] As can be seen in Table 1, PoseC3D showed the best performance at NTU-RGB+D 60 and 120, but we analyzed the experimental results excluding PoseC3D because it uses pre-training, and the estimated poses were optimized for the model rather than the annotated poses. Therefore, we can compare the SoTA models tested under mainly similar conditions.

[0246] In the case of NTU-RGB+D 60 XSub, the STAR++ Transformer (20) showed slightly lower performance than the VPN and InfoGCN models. This is because the STAR++ Transformer (20) uses only 16 frames and 2D skeleton data, whereas the VPN uses 64 frames and 3D skeleton data for training. Therefore, it was determined that it could learn richer features than the STAR++ Transformer (20) and had slightly higher performance. However, for the more complex NTU-RGB+D 120 dataset, the STAR++ Transformer (20) improved accuracy by 4.6% for XSub and 5.1% for XSet compared to the VPN. For InfoGCN, which uses only skeletons, it showed 0.3% higher accuracy than STAR++ applied only to Xsub, but for XView, which has a difference in camera view, it showed 0.5% lower accuracy than the STAR++ Transformer (20). However, with NTU-RGB+D 120, STAR++ showed an accuracy improvement of 1.1% for Xsub and 1.7% for Xset compared to InfoGCN.

[0248] In the results for NTU-RGB+D 60 and NTU-RGB+D 120 XSub and XSet, the STAR++ Transformer (20) showed the highest score among the SoTA methods, excluding PoseC3D, despite lacking training. From the results, it can be confirmed that the proposed STAR++ algorithm is a very competitive algorithm for motion recognition. KA-AGTN, which tokenizes Transformer-based skeletal information for motion recognition similar to the STAR++ Transformer (20), was found to have an accuracy of 4.8% lower for XSub and 4.9% lower for XSet than STAR++. Additionally, the accuracy of the STAR++ Transformer (20) increased evenly across NTU-RGB+D 60 and 120 compared to the base network, STAR-Transformer. These results demonstrate that the tokenization method of the STAR++ Transformer (20) can more flexibly combine each multimode feature.

[0250] Results of the Penn-Action dataset

[0251] Table 2 shows the results of a performance comparison between the SoTA model and the proposed STAR++ transformer (20) using the Penn-Action dataset. As a SoTA method, 3D Deep (C. Cao, Y. Zhang, C. Zhang and H. Lu, "Body joint guided 3-d deep convolutional descriptors for action recognition," IEEE Trans. on Cybernetics, vol. 48, no. 3, pp.1095-1108, Mar. 2017, doi: 10.1109 / TCYB.2017.2756840.) PoseMap, Multi CNN (DC Luvizon, D. Picard and H. Tabia, "2d / 3d pose estimation and action recognition using multitask deep learning," in Proc. CVPR, Salt Lake City, UT, USA, 2018, pp. 5137-5146.), HDM-BG (R. Zhao, W. Xu, H. Su and Q. Ji, "Bayesian hierarchical dynamic model for human action recognition," in Proc. CVPR, Long Beach, CA, USA, 2019, pp 7733-7742.), Pr-VIPE (J. Sun, J. Zhao, L.-C. Chen, F. Schroff, H. Adam and T. Liu, "Viewinvariant probabilistic embedding for human pose," in Proc. ECCV, Virtual, 2020, pp 53-70.), UNIK (D. Yang, Y. Wang, A. Dantcheva, L. Garattoni, G. Francesca and F.It was compared with Bremond, "Unik: A unified framework for real-world skeleton-based action recognition," in Proc. BMVC, Virtual, 2021, pp. 1-13.) and STAR Transformer.

[0253]

[0255] Pr-VIPE and UNIK, single-model approaches based on GNNs that use skeletal information, showed accuracy of 97.5% and 97.9%, respectively, as pre-trained models. However, these methods showed relatively lower accuracy compared to Multi CNN, STAR-Transformer, and STAR++ Transformer (20) models that use a multi-modal approach without pre-training. From the experimental results, we can see that performance is better when using a multi-modal method than a single-modal approach. The accuracy of the proposed STAR++ Transformer (20) model was 99.3%, which is 0.6% higher than the basic STAR-Transformer model and achieved SoTA performance on the Penn-Action dataset. These results show that each module included in the STAR++ Transformer (20), an extended version of the STAR-Transformer, operates efficiently.

[0257] D. Model Analysis and Ablation Study

[0259] The performance of the proposed STAR++ algorithm was experimentally verified. All experiments were performed using the Penn-Action dataset.

[0261] Performance evaluation of IAttn and D3D

[0262] We tested how the cross-attention mechanism (IAttn) of the interval attention module (230), which is a core module of STAR++, and the token selection (D3D) of the D3D token selection module (210) affect model training. Table 3 shows the results of a performance comparison when each module is used together and when used individually.

[0264]

[0266] Since the STAR++ Transformer (20) is a structure based on the STAR-Transformer, using the two modules is equivalent to using the basic STAR-Transformer. As shown in Table 3, when only D3D was used, the accuracy increased by 0.2% compared to when neither module was used. Similarly, when IAttn was used alone, the accuracy increased by 0.4% compared to when neither module was applied. Finally, when all modules were used together, the highest performance (99.3%) was achieved. The results show that the modules of each STAR++ Transformer (20) have complementary characteristics and can further improve performance when combined harmoniously.

[0268] Performance comparison between single and multi-modal approaches

[0269] To determine how the cross-modal mechanism affects single and multi-modal tokens, we further compared performance when using and not using multi-class tokens. As shown in Table 4, among single-modal features, performance was superior when using only RGB compared to applying only Skeleton, regardless of whether multi-class tokens were used. Additionally, performance was higher when using multi-class tokens than when not using them. When using multi-class tokens, multi-modal performance was 0.8% to 1.1% higher than with single-modal tokens. From these results, it can be seen that multi-class tokens not only improve model performance but also help enhance performance by flexibly combining multi-modal tokens.

[0271]

[0273] Performance comparison based on interval terms

[0274] Additional experiments were conducted to verify the effect of the interval term used in IAttn of the STAR++ Transformer (20) on learning. As shown in Table 5, it was found that the STAR++ Transformer (20) exhibited the best performance when using the interval term I={1,4,8}. Since the Transformer focuses on learning from local and global features as the layers become deeper, it was found that the STAR++ Transformer (20) exhibited the highest performance when the interval term increased according to the number of layers. Because the STAR++ Transformer (20) essentially inherited these characteristics, it was able to learn global features as the interval term gradually increased and the layers became deeper.

[0276]

[0278] Therefore, it was found that a structure with gradually increasing interval terms exhibited better performance than a structure with gradually decreasing interval terms. Based on the experimental results, the proposed STAR++ transformer (20) model used interval terms {1, 4, 8} in all experiments performed.

[0280] Comparison of complexity

[0281] In this invention, a D3D token selection method is proposed as an efficient token selection approach. To demonstrate the efficiency of the proposed D3D token selection, the number of parameters and computational load of the basic STAR-Transformer and the STAR++ Transformer (20) were compared. As can be seen in Table 6, the number of parameters used in the STAR++ Transformer (20) increased by nearly twofold, but the number of GFLOP operations decreased by about 1.06 times. Although the model size increased, the proposed STAR++ Transformer (20) was proven to be more efficient than the STAR-Transformer in terms of reducing the number of GFLOP operations and improving performance.

[0283]

[0285] Visualization

[0286] Attention scores were visualized to determine which part of the spatiotemporal features of the input frame is focused by the proposed IAttn and D3D token selection approach for behavior recognition.

[0288] FIG. 10 is a diagram illustrating the visualization results of the NTU-RGB+D 60 and Penn behavior datasets using a human behavior recognition system (10) and method using a spatiotemporal cross-attention transformer according to an embodiment of the present invention. That is, it shows a visualized attention map of the frames used for learning. As shown in FIG. 10, in the case of a single person, interest (attention) was high at locations where the movement changed significantly, and in the case of multiple people, interest was high centered on the person with large movements. The attention scores were concentrated on the position of the hand during a handshake, the trajectory of the hand movement when removing glasses, and the position of the face. Also, in the case of a tennis serve and a pull-up, the attention scores were high centered on the person who moved significantly. As such, it can be seen that the attention of the global grid tokens representing RGB is concentrated on the intensity of the movement and the object.

[0290] FIG. 11 is a diagram illustrating the visualization of attention scores for joint map tokens representing skeletal information in a human behavior recognition system (10) and method using a spatiotemporal intersection attention transformer according to an embodiment of the present invention. Four frames with high attention scores for each joint were extracted and visualized. In the diagram, the larger the attention score, the larger the red circle, and the smaller the attention score, the smaller the circle. As shown in FIG. 11, when there is a large difference in the action of each joint, the position of the joint changes significantly, resulting in a high attention score. In particular, the attention score was high when the position of the joint changed significantly or the position of the joint became closer as time passed. For example, in the case of the ball-throwing motion indicated in the first row, the attention score was high when the body was crouched before throwing, when the head height was lowered, and when the position of the foot or hand changed significantly after throwing the ball. In addition, the attention score was high for the shoulders and knees, which moved significantly from a crouched position to an upright position when squatting. In the 4th row badminton motion, the position of the entire skeleton changed because the player was hit by the shuttlecock while jumping, and in the 3rd frame, the total joint score increased. From these results, it can be confirmed that the proposed STAR++ Transformer (20) learns while accurately identifying the spatiotemporal characteristics of the action video.

[0292] As described above, according to the human behavior recognition system (10) and method using the improved spatiotemporal cross-attention transformer proposed in the present invention, behavior recognition is performed by learning video frames and human skeletal features together, and data of heterogeneous domains can be effectively learned by flexibly combining image and skeletal-based data through multi-representation learning and applying multi-class tokens, and the spatiotemporal features can be learned together through the transformer (STAR++ transformer (20)) to which the improved spatiotemporal cross-attention is applied, thereby improving the behavior recognition performance.

[0294] In addition, according to the human behavior recognition system (10) and method using the improved spatiotemporal cross-attention transformer proposed in the present invention, the STAR++ transformer (20) is configured with a plurality of encoder layers including an interval attention module (230) that creates two token groups by dividing them according to a preset interval term based on the time dimension and learns behavioral changes by performing a cross-attention mechanism between the two token groups, thereby enabling the gradual learning of differences between large and small features of the time axis according to depth, and the network can be simplified and computational costs reduced by integrating into a single encoder form rather than an encoder-decoder structure.

[0296] Meanwhile, the present invention may include a computer-readable medium comprising program instructions for performing operations implemented by various communication terminals. For example, the computer-readable medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory.

[0298] Such a computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. In this case, the program instructions recorded on the computer-readable medium may be those specifically designed and configured to implement the present invention, or they may be those known and available to those skilled in the art of computer software. For example, they may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0300] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0302] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0303] 10: Human Behavior Recognition System 20: STAR++ Transformers 100: Data processing unit 200: Behavior Recognition Unit 210: D3D Token Selection Module 220: Total Attention Module 230: Spacing Attention Module S10: Step where the D3D token selection module generates keys and values ​​from the input token S20: The step in which the total attention module performs the total spacetime attention mechanism S30: A step of generating two token groups by dividing the output of the entire attention module according to a preset interval term based on the time dimension. S40: Step where the D3D token selection module generates keys and values ​​from two token groups. S50: Step in which the interval attention module performs the cross-attention mechanism between two token groups S100: Step of constructing input tokens from input video and skeletal data S200: Step of inputting the input token into the STAR++ Transformer to estimate the final action label.

Claims

Claim 1 A behavior recognition system that recognizes human behavior using input video and skeleton data, comprising: a data processing unit (100) that forms input tokens from the input video and skeleton data; The human behavior recognition system (10) using an improved spatio-temporal cRoss attention transformer is configured to include a behavior recognition unit (200) that inputs the above input tokens into an improved spatio-temporal cRoss attention (STAR++) transformer to estimate a final behavior label, wherein the STAR++ transformer (20) is composed of a plurality of encoder layers including an interval attention (IAttn) module that performs an attention mechanism, divides the above input tokens according to a preset interval term based on the time dimension to create two token groups, and performs an intersection attention mechanism between the two token groups to learn behavior changes, and wherein the data processing unit (100) extracts global features and local features from the frames of the above input video, combines a global grid token that tokenizes the above global features with a joint map token that tokenizes the above local features and skeletal data, and includes a multi-class token to form the input tokens. Claim 2 A human behavior recognition system (10) using an improved spacetime intersection attention transformer, wherein, in claim 1, the interval term is preset for each encoder layer, and is set so that the value increases as the layer deepens. Claim 3 delete Claim 4 A human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer, wherein, in the first paragraph, the data processing unit (100) is characterized by tokenizing a joint heatmap generated using the joint coordinates of the skeletal data by combining it with the local features to generate the joint map token. Claim 5 A human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer, wherein, in claim 1, the data processing unit (100) combines the global grid token and the joint map token, and connects the multi-class token between both ends of the entire token and the global grid token and the joint map token to form the input token. Claim 6 A human behavior recognition system (10) using an improved spatio-temporal cross-attention transformer, wherein the encoder layer comprises: a full attention module (220) that performs a full spatio-temporal attention (FAttn) mechanism for all tokens constituting the input tokens; and an interval attention module (230) that divides the output of the full attention module (220) according to a preset interval term based on the time dimension to create two token groups and performs a cross-attention mechanism between the two token groups. Claim 7 A human behavior recognition system (10) using an improved spatiotemporal intersection attention transformer, wherein the STAR++ transformer (20) further includes a D3D token selection module (210) that extracts global grid tokens from input tokens, combines offsets and reference points to generate final coordinates, and generates keys and values ​​from the final coordinates. Claim 8 In claim 7, the encoder layer comprises: (a) a step in which the D3D token selection module (210) generates a key and a value from the input token; (b) a step in which the total attention module (220) performs a total spatiotemporal attention mechanism using the key and value generated by the D3D token selection module (210); (c) a step in which the output of the total attention module (220) is divided according to a preset interval term based on the time dimension to generate two token groups; (d) a step in which the D3D token selection module (210) generates a key and a value from the two generated token groups; and (e) a step in which the interval attention module (230) performs a cross-attention mechanism between the two token groups generated in step (c) using the key and value generated in step (d). This characterizes a human behavior recognition system (10) using an improved spatiotemporal cross-attention transformer. Claim 9 A behavior recognition method for recognizing human behavior using input video and skeleton data, wherein each step is performed on a computer, comprising: (1) a step of forming an input token from the input video and skeleton data; The method for recognizing human behavior using an improved spatio-temporal cRoss attention transformer is configured to include the step of inputting the input token into an improved spatio-temporal cRoss attention (STAR++) transformer to estimate the final behavior label, wherein the STAR++ transformer (20) is composed of a plurality of encoder layers including an interval attention (IAttn) module that performs an attention mechanism, divides the input token according to a preset interval term based on the time dimension to create two token groups, and performs an interval attention mechanism between the two token groups to learn behavioral changes, wherein in step (1), global features and local features are extracted from the frames of the input video, and a global grid token that tokenizes the global features is combined with a joint map token that tokenizes the local features and skeletal data, and the input token is constructed by combining the two tokens including a multi-class token. Claim 10 A human behavior recognition method using an improved spatiotemporal intersection attention transformer, characterized in that, in claim 9, the interval term is preset for each encoder layer, and is set so that its value increases as the layer deepens. Claim 11 delete Claim 12 A method for recognizing human behavior using an improved spatiotemporal cross-attention transformer, wherein in step (1) above, a joint heatmap generated using the joint coordinates of the skeletal data is combined with the local features to tokenize and generate the joint map token. Claim 13 A method for recognizing human behavior using an improved spatiotemporal cross-attention transformer, characterized in that, in step (1) above, the global grid token and the joint map token are combined, and the multi-class token is connected between both ends of the entire token and the global grid token and the joint map token to form the input token. Claim 14 A method for recognizing human behavior using an improved spatio-temporal cross-attention transformer according to claim 9, wherein the encoder layer comprises: a full attention module (220) that performs a full spatio-temporal attention (FAttn) mechanism for all tokens constituting the input tokens; and an interval attention module (230) that divides the output of the full attention module (220) according to a preset interval term based on the time dimension to create two token groups and performs a cross-attention mechanism between the two token groups. Claim 15 A human behavior recognition method using an improved spatiotemporal intersection attention transformer, wherein the STAR++ transformer (20) further includes a D3D token selection module (210) that extracts global grid tokens from input tokens, combines offsets and reference points to generate final coordinates, and generates keys and values ​​from the final coordinates. Claim 16 In claim 15, the encoder layer comprises: (a) a step in which the D3D token selection module (210) generates a key and a value from the input token; (b) a step in which the total attention module (220) performs a total spatiotemporal attention mechanism using the key and value generated by the D3D token selection module (210); (c) a step in which the output of the total attention module (220) is divided according to a preset interval term based on the time dimension to generate two token groups; (d) a step in which the D3D token selection module (210) generates a key and a value from the two generated token groups; and (e) a step in which the interval attention module (230) performs a cross-attention mechanism between the two token groups generated in step (c) using the key and value generated in step (d). A method for recognizing human behavior using an improved spatiotemporal cross-attention transformer.

Images