System and method for object enhanced action recognition

Abstract

A computer-implemented method of object enhanced motion recognition and an object enhanced motion recognition system may be provided, the method comprising processing one or more initial image tile tokens to extract one or more human tokens and one or more object tokens; applying the extracted one or more human tokens and the extracted one or more object tokens and the one or more initial image tile tokens to a first cross attention mechanism to obtain a set of processed image tile tokens containing object guidance information; enumerating to construct a set of human-object (HO) tokens, the constructed HO tokens each including information associated with a visual relationship and information associated with a spatial relationship between a human token and an object token; applying the constructed HO token and the processed image block token containing the object guidance information to a second cross attention mechanism to obtain a set of resulting processed HO tokens; applying a feature aggregation process to the resulting processed HO tokens to synthesize features of the resulting processed HO tokens of the M groups into a single feature representation; obtaining a prediction result based on the group of processed image block tokens containing the object guidance information; obtaining an action recognition prediction result based on a result of the feature aggregation process and a prediction result of the group of processed image block tokens containing object guidance information; and outputting the motion recognition prediction result for use.

Description

Technical Field

[0001] This disclosure generally relates to an object augmentation action recognition system and an object augmentation action recognition method. Background Technology

[0002] With the development of deep learning technology and the proliferation of video datasets, the field of video-based action recognition / determination has seen significant improvements. Conventionally, recognition methods typically focus on analyzing global visual features or human poses to identify the actions of one or more people from video frames. However, it has since been recognized that supplementary information can be incorporated to enrich the contextual understanding of actions. For example, objects present in a video frame can provide important clues / information, thus offering additional attentional information that can significantly improve the accuracy of action recognition / determination.

[0003] Several studies have investigated the effects of incorporating objects into action recognition models. These studies include: J. Materzynska et al., Something-else: Compositional action recognition with spatial-temporal interaction networks, Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 1049-1059; G. Radevski et al., Revisiting spatiotemporal layouts for compositional action recognition, arXiv preprint arXiv:2111.01936 (2021); R. Herzig et al., Object-region video transformers, Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. Recognition, [IEEE / CVF Conference on Computer Vision and Pattern Recognition Proceedings], 2022, pp. 3148-3159; C. Zhang et al., Is an object-centric video representation beneficial for transfer?, in: Proceedings of the Asian Conference on Computer Vision, 2022, pp. 1976-1994; and X. Zhou et al., How can objects help action recognition?, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2353-2362.

[0004] Other studies have attempted to integrate object information into action-related tasks, such as action recognition and action detection. These studies include: G. Elsayed et al., Savi++: Towards end-to-end object-centric learning from real-world videos, Advances in Neural Information Processing Systems 35(2022)28940–28954; F. Locatello et al., Object-centric learning with slotattention, Advances in Neural Information Processing Systems 33(2020)11525–11538; A. Shamsian et al., Learning object permanence from video, Computer Vision–ECCV 2020:16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16 [Computer Vision – ECCV 2020: The 16th European Conference, Glasgow, UK, 23-28 August 2020, Proceedings, Part XVI 16], Springer, 2020, pp. 35-50; C.-Y. et al., Towards long-form video understanding, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1884-1894; and X. Wang et al., Videos as space-time region graphs, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 399-417.

[0005] However, existing research and critical analysis of this model reveal a problem or a common shortcoming: a lack of consideration for the complex interactions between objects and humans in the scene and / or the relationships between certain objects and actions.

[0006] The inventors recognize that the lack of consideration in the above discussion often leads to the introduction of action-irrelevant information or the injection of irrelevant object information, thereby impairing the results of feature extraction and the overall model performance.

[0007] For current proximity technologies, such as those that determine interactions rather than actions, Human Object Interaction (HOI) tasks are typically studied as traditional computer vision (CV) tasks. With the development of Transformer models in the context of deep learning, HOI detection methods are generally divided into two main categories: two-stage methods and single-stage methods.

[0008] Single-stage HOI detection methods are generally known for their efficiency and simplicity. These methods directly predict interaction triples, typically bypassing the relatively more complex post-processing steps used in two-stage methods. Initially, single-stage methods focused on predicting interactions as sets of keypoints or detecting interaction regions via anchor boxes. These early methods attempted to directly capture the nuances of interactions but faced challenges such as semantic ambiguity and occlusion sensitivity. Single-stage methods also include end-to-end transformer-based approaches, which leverage self-attention mechanisms to comprehensively model complex relationships between entities within an image. This approach integrates human-object pair detection and interaction classification into a unified framework and can improve context capture and reduce the influence of irrelevant instances. The inventors recognized that using transformers allows for a more granular understanding of interactions.

[0009] On the other hand, for typical two-stage HOI detection methods, potential human-object (HO) pairs are first detected, and then their interactions are analyzed. This approach typically involves cropping one or more humans and objects from the bounding box and extracting features such as appearance, spatial configuration, and pose to predict the interactions between the pairs.

[0010] Typically, the first stage of a two-stage approach uses an object detection model to identify objects and people in an image. This involves a model pre-trained on datasets such as MS-COCO (e.g., T.-Y. Lin et al., Microsoft coco: Common objects in context, ComputerVision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer, 2014, pp. 740-755) and a HOI-specific dataset such as HICO-DET (e.g., Y.-W. Chao et al., Hico: A benchmark for recognizing human-object interactions). Fine-tuning was performed on inimages [Hico: a benchmark for recognizing human-object interactions in images, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1017-1025] to improve detection accuracy.

[0011] For the second stage, the interaction type is typically predicted by analyzing the extracted features (including appearance, spatial location, pose, and semantic information). This is taught, for example, in the following literature: S. Qi et al., Learning human-object interactions by graph parsing neural networks, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 401-417.

[0012] With advancements in feature extraction techniques, such as the use of convolutional neural networks (CNNs) like ResNet (as taught in: K. He et al., Deep residual learning for image recognition, Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778) and deformable ConvNet (as taught in: J. Dai et al., Deformable convolutional networks, Proceedings of the IEEE international conference on computer vision, 2017, pp. 764-773), the second stage can be improved, and different feature types can be explored to more accurately predict interactions. In other research, graph networks have been introduced to capture high-level structures and relationships in video scenes. These graph networks can be visual graphs that associate appearance features, semantic graphs that rely on word embeddings, or combined visual graphs that include both visual graphs and semantic cues.

[0013] Furthermore, other studies have attempted to incorporate transformer-based models into two-stage methods / frameworks. PViC (as taught in the following paper: FZZhang et al., Exploring predicate visual context in detecting of human-object interactions, in: Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 10411-10421) enhances predicate visual context by spatially guiding location embeddings through improved query design and integration boxes, addressing some limitations of traditional two-stage methods. GEN-VLKT (as taught in the following literature: Y. Liao et al., Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20123-20132) addresses some of the shortcomings of conventional query-driven detectors by introducing guided embedding networks. This network can simplify human-object associations and enhance interaction understanding through visual language knowledge transfer from pre-trained models. These models include CLIP (as taught in the following literature: A. Radford et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748-8763).ViPLO (as taught in the following literature: J. Park et al., Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17152-17162) can integrate a visual Transformer backbone network with a pose-conditioned self-loop graph, can emphasize human perception processes in HOI detection, and can achieve state-of-the-art results on public benchmarks.

[0014] However, the inventors recognized that the two-stage approach in existing technologies could not be used in action recognition tasks due to the lack of interactive-level annotations.

[0015] Therefore, there is a need for an object augmentation action recognition system and an object augmentation action recognition method that attempt to solve at least one of the above problems. Summary of the Invention

[0016] According to one aspect of this disclosure, a computer-implemented method for object-enhanced action recognition is provided, the method comprising obtaining one or more initial image patch tokens from one or more video frames. The process involves: processing the initial image patch tokens to extract one or more human tokens and one or more object tokens; applying the extracted human tokens, object tokens, and initial image patch tokens to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object guidance information; enumerating the extracted human tokens and object tokens to construct a set of human-object (HO) tokens, each HO token including information associated with visual relationships and information associated with spatial relationships between human and object tokens; applying the constructed HO tokens and processed image patch tokens containing object guidance information to a second cross-attention mechanism to obtain a set of resulting processed HO tokens; applying a feature aggregation process to the resulting processed HO tokens to synthesize the features of M sets of resulting processed HO tokens into a single feature representation; obtaining a prediction result based on the set of processed image patch tokens containing object guidance information; obtaining an action recognition prediction result based on the result of the feature aggregation process and the prediction result based on the set of processed image patch tokens containing object guidance information; and outputting the action recognition prediction result for use.

[0017] The steps of processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens may include using object metadata information to extract the one or more human tokens and one or more object tokens, the object metadata information being based on applying bounding box coordinates and object categories from object annotation data to the one or more video frames.

[0018] The step of processing the one or more initial image block tokens to extract one or more human tokens and one or more object tokens may further include sorting the extracted one or more human tokens and one or more object tokens to place the one or more human tokens in front of the one or more object tokens.

[0019] The step of enumerating one or more extracted human tokens and one or more extracted object tokens to construct a set of HO tokens may further include embedding information associated with visual relationships using a splicing process and a positional encoding process.

[0020] The step of enumerating one or more extracted human tokens and one or more extracted object tokens to construct a set of HO tokens may further include using spatial embedding data and another stitching process to embed information associated with spatial relationships.

[0021] The steps of the applied feature aggregation process may include combining different obtained processed HO tokens into M groups, where each group represents an action-level feature trained using action annotations; and aggregating the obtained processed HO tokens using weights applied to each obtained processed HO token.

[0022] The step of applying the feature aggregation process to the resulting processed HO tokens may further include selecting the most decisive processed HO tokens for action recognition.

[0023] The step of outputting the motion recognition prediction result for use may include: transmitting the motion recognition prediction result to a motion device; and causing the motion device to activate a motion based on the motion recognition prediction result.

[0024] According to another aspect of this disclosure, an object-enhanced action recognition system is provided. This system includes an input module for obtaining one or more initial image patch tokens from one or more video frames; an object-enhanced relation-aware action recognition (ORA) module coupled to the input module, the ORA module including an object-guided attention (OGA) module for processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens; the OGA module further for applying the extracted one or more human tokens, the extracted one or more object tokens, and the one or more initial image patch tokens to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object-guided information; and a dynamic interactive attention (DIA) module coupled to the OGA module, the DIA module for enumerating the extracted one or more human tokens and the extracted one or more object tokens to construct a set of human-object (HO) tokens, the constructed HO... Each of the O tokens includes information associated with visual relationships and information associated with spatial relationships between human tokens and object tokens. The DIA module is further used to apply the constructed HO tokens and processed image patch tokens containing object guidance information to a second cross-attention mechanism to obtain a set of processed HO tokens. A feature aggregation module is used to apply a feature aggregation process to the processed HO tokens to synthesize the features of M sets of processed HO tokens into a single feature representation. A classification head is used to obtain a prediction result based on the set of processed image patch tokens containing object guidance information. A prediction module is used to obtain an action recognition prediction result based on the result of the feature aggregation process and the prediction result based on the set of processed image patch tokens containing object guidance information. The prediction module is further used to output the action recognition prediction result for use. An action device coupled to the ORA module is arranged to activate an action based on the action recognition prediction result.

[0025] The object augmentation motion recognition system may further include the OGA module for using object metadata information to extract the one or more human tokens and the one or more object tokens, the object metadata information being arranged by the OGA module to apply bounding box coordinates and object categories from object annotation data to the one or more video frames.

[0026] The object augmentation action recognition system may further include the OGA module for sorting the extracted one or more human tokens and the extracted one or more object tokens in order to place the one or more human tokens in front of the one or more object tokens.

[0027] The object-enhanced action recognition system may further include the DIA module for embedding information associated with visual relationships using a stitching process and a positional encoding process.

[0028] The object-enhanced action recognition system may further include the DIA module for embedding information associated with spatial relationships using spatial embedding data and another stitching process.

[0029] The object-enhanced action recognition system may further include a feature aggregation module for combining different obtained processed HO tokens into M groups, where each group represents an action-level feature trained using action annotations; and for aggregating the obtained processed HO tokens using weights applied to each obtained processed HO token.

[0030] The object-enhanced action recognition system may further include a feature aggregation module for performing the selection of the most decisive processed HO token for action recognition.

[0031] The object-enhanced motion recognition system may further include a prediction module for transmitting the motion recognition prediction result to the motion device; and for enabling the motion device to activate the motion based on the motion recognition prediction result.

[0032] According to another aspect of this disclosure, a non-transitory tangible computer-readable storage medium is provided, having stored thereon software instructions that, when executed by a computer processor of an object augmentation action recognition system, cause the computer processor to perform a computer-implemented method of object augmentation action recognition by performing the following steps: obtaining one or more initial image patch tokens from one or more video frames; processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens; applying the extracted one or more human tokens and the extracted one or more object tokens, along with the one or more initial image patch tokens, to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object guidance information; and enumerating the extracted one or more human tokens and the extracted object tokens... One or more object tokens are used to construct a set of human-object (HO) tokens, each of which includes information associated with visual relationships and information associated with spatial relationships between human tokens and object tokens; the constructed HO tokens and processed image patch tokens containing object guidance information are applied to a second cross-attention mechanism to obtain a set of resulting processed HO tokens; a feature aggregation process is applied to the resulting processed HO tokens to synthesize the features of M sets of resulting processed HO tokens into a single feature representation; a prediction result is obtained based on the set of processed image patch tokens containing object guidance information; an action recognition prediction result is obtained based on the result of the feature aggregation process and the prediction result based on the set of processed image patch tokens containing object guidance information; and the action recognition prediction result is output for use.

[0033] The computer-implemented method for enhancing action recognition of this object can be as described in the foregoing aspects and statements. Attached Figure Description

[0034] Exemplary embodiments of this disclosure will be better understood and will be apparent to those skilled in the art from the following written description, by way of example only and in conjunction with the accompanying drawings, in which:

[0035] Figure 1 This is a schematic block diagram used to illustrate the object-enhanced action recognition system in an exemplary embodiment.

[0036] Figure 2 This is a schematic block diagram illustrating a computer-implemented method for object-enhanced relationship-aware action recognition in an exemplary embodiment.

[0037] Figure 3 This is a schematic diagram illustrating a subsystem for constructing human-object (HO) tokens in an exemplary embodiment.

[0038] Figure 4AIt is a schematic diagram illustrating the typical Transformer model structure.

[0039] Figure 4B This is a schematic diagram illustrating the structure of an example Transformer model in an exemplary embodiment.

[0040] Figure 5 This is a schematic block diagram illustrating the method for object-enhanced action recognition in an exemplary embodiment.

[0041] Figure 6 This is a schematic diagram of a computer system suitable for implementing exemplary embodiments. Detailed Implementation

[0042] Exemplary embodiments may provide an object-enhanced action recognition system and an object-enhanced action recognition method. In exemplary embodiments, action recognition may be enhanced through one or more processes to provide information about the object to enhance action recognition and / or to provide information about the relationship between the object and the human to enhance action recognition.

[0043] Figure 1 This is a schematic block diagram illustrating an object-enhanced action recognition system 100 in an exemplary embodiment. System 100 includes an input module 102 arranged / configured to obtain one or more initial image block tokens from one or more video frames. System 100 further includes an object-enhanced relation-aware action recognition (ORA) module 104 coupled to the input module 102.

[0044] In an exemplary embodiment, ORA module 104 includes an Object Guided Attention (OGA) module 106, which is arranged / configured to process one or more initial image patch tokens to extract one or more human tokens and one or more object tokens. OGA module 106 is further arranged / configured to apply the extracted one or more human tokens, the extracted one or more object tokens, and the one or more initial image patch tokens to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object guidance information. ORA module 104 also includes a Dynamic Interactive Attention (DIA) module 108 coupled to OGA module 106, and DIA module 108 is arranged / configured to enumerate the extracted one or more human tokens and the extracted one or more object tokens to construct a set of HO tokens, each of which includes information associated with a visual relationship and information associated with a spatial relationship between the human token and the object token. DIA module 108 is further arranged / configured to apply the constructed HO tokens and the processed image patch tokens containing object guidance information to a second cross-attention mechanism to obtain a set of resulting processed / updated HO tokens.

[0045] ORA module 104 further includes a feature aggregation module 110, which is arranged / configured to apply a feature aggregation process to the resulting processed HO tokens to synthesize the features of M groups of processed HO tokens into a single feature representation. ORA module 104 further includes a classification head or classification head module 111, which is arranged / configured to obtain a prediction result based on the group of processed image block tokens containing object guidance information. ORA module 104 also includes a prediction module 112, which is arranged / configured to obtain an action recognition prediction result based on the result of the feature aggregation process and the prediction result based on the group of processed image block tokens containing object guidance information. Prediction module 112 is further arranged / configured to output the action recognition prediction result for use.

[0046] In an exemplary embodiment, system 100 further includes an action device 114 coupled to ORA module 104. The action device 114 is arranged / configured to activate an action based on the action recognition prediction result.

[0047] In an exemplary embodiment, the OGA module 106 may be further arranged / configured to use object metadata information to extract the one or more human tokens and the one or more object tokens, the object metadata information being arranged based on the OGA module 106 to apply bounding box coordinates and object categories from object annotation data to the one or more video frames.

[0048] In an exemplary embodiment, the OGA module 106 may be further arranged to / configured to sort the extracted one or more human tokens and the extracted one or more object tokens to place the one or more human tokens in front of the one or more object tokens.

[0049] In an exemplary embodiment, the DIA module 108 may be arranged / configured to embed information associated with visual relationships using a stitching process and a positional encoding process. In an exemplary embodiment, the DIA module 108 may be arranged / configured to embed information associated with spatial relationships using spatial embedding data and another stitching process.

[0050] In an exemplary embodiment, the feature aggregation module 110 may be arranged / configured to combine different obtained processed HO tokens into M groups, where each group represents an action-level feature trained using action annotations; and to aggregate the obtained processed HO tokens using weights applied to each obtained processed HO token. The feature aggregation module 110 may be arranged / configured to perform the selection of the most decisive obtained processed HO token for action recognition.

[0051] In an exemplary embodiment, the prediction module 112 may be further arranged / configured to transmit the action recognition prediction result to the action device 114 and cause the action device 114 to activate the action based on the action recognition prediction result.

[0052] In an exemplary embodiment, the motion recognition prediction result may technically influence the motion device 114 external to the ORA module 104. For example, in automotive applications, the motion recognition prediction result can activate possible corrective / preventive actions. For instance, if a person's action is identified as chasing an object across the street, the motion device 114 can activate an action based on the motion recognition prediction result, which may include, for example, but not limited to, activating the vehicle's braking mechanism to avoid impacting the person. Besides automotive applications, the motion recognition prediction result can also be used in other applications, such as, but not limited to, aerospace applications, engineering equipment operation applications, etc. The inventors recognize that the motion recognition prediction result is useful in applications where user or other distraction may lead to / induce accidents. As another example, it can be specified that one or more alarm mechanisms can be activated based on the motion recognition prediction result. As another example, it can be specified that an emergency display of the motion recognition prediction result can be activated based on the motion recognition prediction result. This can be achieved by visually displaying it to the operator / user on a screen.

[0053] In an exemplary embodiment, the ORA model can effectively enhance / improve action recognition by using relation-aware attention and object-guided insights to more accurately identify and classify actions based on dynamic interactions between humans and objects. As described above, by fusing object and relational information from the final layer of the Visual Transformer (ViT) model stack, image patch tokens can progressively and effectively acquire this guiding information in the Object-Guided Attention (OGA) branch, while human-object (HO) tokens can obtain a global representation based on general action-level information in the Dynamic Interaction Attention (DIA) branch.

[0054] In one or more of the exemplary embodiments described, the ViT backbone network is used to obtain image patch tokens P that focus on global action information. Human tokens H and object tokens O are extracted from P using bounding box coordinates and category information via the RoIAlign process. H and O can then be used to interact with P to progressively shift attention to object-related regions. The extracted human and object tokens enumerate all-person-object (HO) pairs and are then combined with processed multidimensional location embeddings to construct HO tokens HO. HO can be used as a query for cross-attention with P, thereby in turn perceiving global action information.

[0055] Furthermore, by clustering different HO tokens under action-level annotation supervision, in one or more exemplary embodiments described, the feature aggregation module can select one or more of the most decisive HO tokens for action recognition. A weighted average of the action recognition result and the interaction result is used to obtain the action recognition prediction result. In the description, "interaction result" refers to the prediction result (or action) related to human-object interaction from the DIA module / branch. For example, see below. Figure 2 Box 232.

[0056] Figure 2 This is a schematic block diagram illustrating a computer-implemented method for object-enhanced relation-aware action recognition in an exemplary embodiment. The method can be implemented using a deep learning or machine learning model called an object-enhanced relation-aware action recognition (ORA) model. In an exemplary embodiment, the ORA model is implemented as ORA module 202. In an exemplary embodiment, ORA module 202 and... Figure 1 The ORA module 104 is essentially similar in nature and functionality. ORA module 202 includes an Object Guided Attention (OGA) module 204 and a Dynamic Interaction Attention (DIA) module 206. ORA module 202 further includes a feature aggregation module 232. Therefore, as... Figure 2 As shown, there are two branches: the OGA branch and the DIA branch. Figure 2 It can be considered as the overall framework of ORA model 202.

[0057] In an exemplary embodiment, OGA module 204 includes blocks 208, 210, 212, 214, 216, 218, and 220. DIA module 206 includes blocks 222, 224, 226, and 230. OGA module 204 is coupled to or includes block 228, which is provided as a classification head. DIA module 206 is further coupled to feature aggregation module 232, or shown as block 232. Block 232 can be used as another classification head. Feature aggregation module 232 is coupled to block 234. In an exemplary embodiment, classification head 228 is also coupled to block 234. In an exemplary embodiment, block 234 can be prediction module 234. In an exemplary embodiment, ORA module 202 further includes prediction module 234.

[0058] At box 208, a Visual Transformer (ViT) model is used to process one or more video frames. In an exemplary embodiment, VideoMAE (a pre-trained ViT model) is used. It should be understood that VideoMAE is a modified version of the ViT model. The VideoMAE model is taught in the following literature: Z. Tong et al., Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, Advances in Neural Information Processing Systems 35(2022)10078–10093; and the specific contents are incorporated herein by reference. In short, VideoMAE uses a masking strategy for efficient pre-training, thereby enhancing learning from a smaller dataset. In an exemplary embodiment, VideoMAE is also used for rapid implementation. In other exemplary embodiments, it should be understood that box 208 may be provided as an input module coupled to ORA module 202. For example, comparison Figure 1 Input module 102.

[0059] In an exemplary embodiment, the OGA branch can introduce object tokens based on VideoMAE (a pre-trained ViT model).

[0060] Given a series of video frames F∈R T×H×W×3 As input, the early layers of the ViT model use a 2×16×16 3D spatiotemporal convolution to generate initial image block tokens (of shape ) at box 210. The image patch tokens are mapped to a d-dimensional matrix. In this description, R refers to a real matrix, and T×H×W×3 defines its dimensions. T refers to the number of frames, H refers to the height of each frame, and W refers to the width of each frame.

[0061] Therefore, as described above, one or more initial image block tokens are obtained from one or more video frames.

[0062] In an exemplary embodiment, in addition to the video frame F, other inputs are utilized. At block 212, using an existing detector, the object metadata B∈{b,c} detected by the detector is used as supplementary input, where b∈R T×N×4 These are the bounding box coordinates relative to the video frame, c∈R T×NN is the object category (or classification), and N is the maximum number of objects specified / predetermined per frame. In the description of this section, the term "object" encompasses (multiple) humans and other objects. Bounding boxes with the human category are placed / ordered first, ahead of all other objects in each current frame, thereby facilitating subsequent HO (human-object) pair matching.

[0063] In an exemplary embodiment, the object-guided ViT layer uses image patch token P (from box 210) from the previous layer and object metadata B (from box 212) from the annotations as input. The first step is to extract the object representation aligned with the dimensions of the image patch token. At box 214, the step of extracting the object representation aligned with the dimensions of the image patch token is taken.

[0064] One approach is to employ a Region of Interest Alignment (RoIAlign) procedure / module to obtain human and object region clipping. The RoIAlign procedure is taught, for example, by K. He et al., Mask r-cnn, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2961-2969; and the details are incorporated herein by reference. At box 214, the RoIAlign procedure / module is followed by a multilayer perceptron (MLP) and pooling layers to obtain complex feature representations and reduce dimensionality to align with image patch tokens. Equation (1) below is provided for the above implementation.

[0065]

[0066] In equation (1), h represents the number of humans in the current frame, so human labels are placed / sorted in front of all other object labels.

[0067] In an exemplary embodiment, for each object instance in each video frame, the human token H and the object token O are combined with a separate image block token P∈R. d They have the same dimensions. Therefore, the overall object tokens, including the human category, are T×N×d vectors. In this exemplary embodiment, since the object token extraction is performed in each object-guided ViT layer, different image patch tokens will generate different human tokens and other object tokens. A schematic representation of the output of box 214 is shown at box 216, where the labeled human tokens are placed / sorted in front of all other object tokens labeled with objects.

[0068] Therefore, as described above, one or more initial image block tokens are processed to extract one or more human tokens and one or more object tokens.

[0069] In an exemplary embodiment, in order to inject object information into the image block token, at block 218, for each l-th layer, the extracted object token H in block 216 is... l and O l Image block token P with box 210 l A multi-head cross-attention (MHCA) process is performed between them. In the description herein, the MHCA process is referred to as the first cross-attention mechanism. The following equation (2) is provided to carry out the above implementation.

[0070] P l ′=MHCA(P l ,concat[P l ],concat[P l H l O l (2) P l H l and O l They are combined as keys and values ​​in the attention mechanism, thereby directing the attention of image block token queries to the region containing the corresponding object.

[0071] At box 218, the first cross-attention mechanism further utilizes a normalization layer and an MLP layer. Equation (3) is provided below to implement the above.

[0072] P l+1 =MLP(LayerNorm(P l ′))+P l (3)

[0073] The normalization layer and MLP layer further refine the image patch labels using the residual structure and feed them into the next layer l+1. According to equation (2), P l ′ provides intermediate variables for the next stage process (associated with the output of the multi-head cross-attention process), and equation (3)P l+1 Image block tokens are provided for the next layer.

[0074] After multi-layer object-guided attention aggregation, an image patch token containing object-guided information is generated at box 220. The inventors recognized that since the ViT backbone network lacks classification tokens, max-pooling layers are performed along the H and W dimensions within each frame to obtain a frame-level global representation for classification. The max-pooling process allows the selection of the maximum value from a range of values. For example, the function maxpool(0.6, 0.5, 0.8) yields a value of 0.8.

[0075] Therefore, as described above, the extracted one or more human tokens, the extracted one or more object tokens, and one or more initial image block tokens are applied to the first cross-attention mechanism to obtain a set of processed image block tokens containing object guidance information.

[0076] After proposing the object-guided attention branch, the inventors recognized that in some cases, incorrect object information might be mistakenly included in action recognition due to misunderstandings of the relationship between objects and actions. Therefore, the inventors proposed introducing interaction information between humans and objects, enabling the Object-Enhanced Relationship-Aware Action Recognition (ORA) module 202 to distinguish the most important / critical objects for action recognition.

[0077] Furthermore, as will be described in more detail below for the Dynamic Interaction Attention (DIA) module 206, the HO (Homologous Object) requires / includes at least one person, which also satisfies the requirements / criteria of action recognition, i.e., because action recognition is for (multiple) human actions. This processing helps the ORA module 202 avoid incorrect judgments of the "no human" category.

[0078] Starting from box 216, using the human token H and the corresponding other object token O extracted for each video frame, at box 224, a human-object (HO) pair table can be generated by enumerating all tokens (see box 222). In some exemplary embodiments, the enumeration process can be performed using enumeration functions provided by platforms such as, but not limited to, Python. The enumeration process or enumeration function combines H tokens with O tokens into pairs. For example, if there are m H tokens and n O tokens, M = m * n non-repeating HO pairs, or M rows of HO pairs, can be provided. First, for box 224, the H tokens and O tokens are concatenated together as a visual embedding of HO pairs with a dimension of 2d (or 2 * d, where d is the dimension of the tokens). As a simplified example, if H is [0.1 0.2 0.3] and O is [0.4 0.5], then the HO is [0.1 0.2 0.3 0.4 0.5].

[0079] The inventors recognized that for most actions, the dominant action category object (excluding humans) is typically spatially close to the human. Therefore, at box 224, a spatial embedding generator is used to generate spatial embedding data (referred to as Pos) reflecting the spatial relationship between the human token and the object token based on bounding box coordinates. The bounding box coordinates can be the coordinates of the bounding box at box 212. For HO tokens, the Pos data can consist of bounding box coordinates, box area, relative box area, box aspect ratio, intersection-over-union (IoU) ratio between the object and the human, and the relative distance between the object and the human. In an exemplary embodiment, multiple linear and activation layers are used to increase the dimensionality of the Pos data to d dimensions.

[0080] To construct the HO token of box 224, a multimodal fusion box was used.

[0081] Figure 3 This is a schematic diagram illustrating a subsystem for constructing human-object (HO) tokens in an exemplary embodiment. Figure 2 The H token 302 and O token 304 of the bounding box 216 are used as input. Data 306, such as bounding box coordinates, is also provided as positioning input. The data 306 is processed by position encoding 308 and provided to one or more encoder layers 310 for processing together with the H token 302 and O token 304. After being processed by one or more encoder layers 310, the H token 302 and O token 304 undergo a concatenation operation 312. The visual embedding, i.e., the concatenation of the H token 302 and O token 304, passes through a fully connected (FC) layer, which includes a normalization layer 314 coupled to the linear layer 316.

[0082] For spatial embedding, data 306 is processed by a multilayer perceptron (MLP) and then normalized through a normalization layer 320 coupled to a linear layer 322. This processing extracts deep features and maintains consistent dimensionality after concatenation. The HO and Pos data undergo another concatenation operation 324 to construct HO tokens 326 with both visual and spatial embedding information. (See comparison...) Figure 2 Box 224.

[0083] Back Figure 2 At box 224, the shape of each constructed HO token is R. d Align it with the previously processed image block token P (the image block token containing object guidance information at box 220). For each video segment V, construct a HO∈R T ×M×d A vector, where M is the maximum number of HO pairs.

[0084] Therefore, as described above, the extracted one or more human tokens and one or more extracted object tokens are enumerated to construct a set of human-object (HO) tokens. As described above, each constructed HO token includes information related to visual relationships and spatial relationships between the human tokens and the object tokens.

[0085] The inventors recognized that HO tokens (HO) might focus more on the interaction between human tokens and object tokens, and might not provide a comprehensive understanding of the entire video frame and video. The inventors further realized that such global information could be obtained from the image block token P generated by ViT.

[0086] At box 226, a second cross-attention mechanism is employed. This process is similar to the first cross-attention mechanism described above with reference equations (2) and (3). It should be understood that the key queries and value sets used for the second cross-attention mechanism (e.g., reference equations (2) and (3)) are different from those used for the first cross-attention mechanism. By concatenating HO (from box 224) and P (from box 220, the processed image block token) as the key and value of the second cross-attention mechanism, the HO group can gradually gain an understanding of the entire video through the cross-attention process, thereby achieving a good balance between local HO attention and global action attention. The resulting processed HO group is indicated at box 230.

[0087] Therefore, as described above, the constructed HO token and the processed image block token containing object guidance information are applied to the second cross-attention mechanism to obtain a set of resulting processed HO tokens.

[0088] The inventors recognized that, in most action recognition datasets, the lack of relational-level annotations for human-object (HO) tokens prevents training with the same level of supervision as traditional human-object interaction (HOI) tasks. Therefore, the inventors recognized and proposed synthesizing the features of M groups of HO tokens into a single feature representation.

[0089] At box 232, a feature aggregation module is provided. Feature aggregation module 232 can synthesize the features of M groups of HO tokens into a single feature representation by combining different HO tokens into groups (e.g., 236), where each group represents an action-level feature trained using action annotations. The different classifications 1 to N of the action-level features (or annotations) are schematically shown in the form of the number 238.

[0090] For a given video segment, the HO token HO∈R T×M×d All tokens are aggregated into an intermediate feature z∈R d Each video segment contains HO = {ho1, ho2, ..., ho...} T×M Then z can be expressed by the following equation (4):

[0091]

[0092] Where i = 1, 2, ..., T×M.

[0093] During the aggregation process, the weight 'a' of each token... i Gated attention is determined by the following equation (5):

[0094]

[0095] Where, w∈Rh V and U∈R h×d is the network parameter, and h is the hidden dimension.

[0096] The inventors recognize that the aggregation provided by feature aggregation module 232 can effectively avoid the inaccuracies typically introduced by average pooling or max pooling processes.

[0097] A simplified example of the feature aggregation process described above is provided below.

[0098] In a video frame, if there are i people and j objects, there are M = i*j HO tokens. The feature aggregation process can add the features (e.g., features x and y) of the M groups of HO tokens to form a new intermediate feature z = w. x *x+w y *y. Each HO token is assigned a weight (w) x or w y Compare equations (4) and (5). This new feature z is an intermediate feature.

[0099] In the training video, there may be humans, telephones (e.g., mobile phones), and infant car seats appearing in one or more frames of the video. Therefore, possible HO tokens could be human-telephone or human-car seat. Action-level features trained using action annotations (e.g., "making a phone call," "texting," or "sitting in an infant car seat") help assign different weights to these tokens. For example, an action-level feature trained using action annotations might lack "sitting in an infant car seat," for example, in the absence of an infant. Therefore, for example, the assigned weights could be 0.99 for the human-telephone token and 0.01 for the human-car seat token. In such an example, the weight of the human-car seat token might always be assigned a relatively low weight, for example, in the absence of an infant. As another simplified example, suppose there are hypothetical features such as "holding a phone," "hand moving towards mouth," and "hand touching the steering wheel." Then, suppose the intermediate feature z can be generally considered as an aggregation or combination of all these hypothetical features, and different weights can be assigned to such features. For example, 0.3 * feature "holding a phone" + 0.4 * feature "hand moving towards mouth" + ... It should be understood that the above description has been simplified for the purpose of understanding only.

[0100] Furthermore, using Pos data, multiple locations of human-object pairs are included in the ORA model. Therefore, during training, human-object pairs that are spatially close to each other are gradually assigned higher weights.

[0101] In the example above using a telephone, the "telephone" was identified as frequently appearing around or near a human in the video frames. Therefore, the human-telephone token was assigned the highest weight among the possible HO tokens and was identified as the most decisive HO token.

[0102] Given the above, the features of all HO tokens are combined to form a single intermediate feature (single feature representation), and HO tokens are assigned different weights, with the HO token having the highest weight being called the "most decisive HO token".

[0103] In an exemplary embodiment, a classification head 228 and a prediction module 234 are provided. A feature aggregation module 232 and the classification head 228 are coupled to the prediction module 234. The prediction module 234 is arranged to combine / process the prediction results from the feature aggregation module 232 and the classification head 228 to provide an action recognition prediction result. The classification head 228 can be implemented using one or more layers including fully connected (FC) layers. Therefore, the classification head 228 includes fully connected layers to, for example, use action recognition from the OGA branch / module 204 based on the feature prediction results. The feature aggregation module 232 can be another classification head. FC layers are employed to facilitate the classification of z into a one-hot prediction vector HO. pred ∈R C The process involves C, where C refers to the number of action categories. A feature aggregation module 232 is provided to use human-object (HO) interactions from the DIA branch / module 206 to predict outcomes.

[0104] Therefore, as described above, the prediction result is obtained based on this set of processed image block tokens containing object guidance information. See classification head 228.

[0105] In an exemplary embodiment, prediction module 234 is arranged to perform weighted summation processing to obtain a prediction result. For example, prediction module 234 may combine / process prediction results from classification head 228 and prediction results from feature aggregation module 232 to provide an action recognition prediction result. For example, prediction module 234 may use a weighted average of action recognition results (from OGA branch / module 204) and interaction (HO) results (from DIA branch / module 206) to calculate the action recognition prediction result. See the output of prediction module 234.

[0106] Therefore, as described above, based on the result of this feature aggregation process and the prediction result based on the group of processed image block tokens containing object guidance information, the action recognition prediction result is obtained.

[0107] In an exemplary embodiment, the output of the motion recognition prediction result from the prediction module 234 can be implemented for use. The output may include transmitting the motion recognition prediction result to an external motion device, and causing the motion device to activate an action based on the motion recognition prediction result. (See also...) Figure 1 Action device 114.

[0108] Below is a simplified example using ORA module 202. The numerical descriptions are for illustrative purposes only. It should be understood that in practical applications, such values ​​are high-dimensional tensors. Such values ​​can come from arrays or matrices, and each value can be generally considered to represent / describe an aspect of the token.

[0109] At box 208, multiple video frames are provided. At box 210, an initial image block token is generated. An example image block token can be a token with the example values ​​[[0.1,0.2], [0.2,0.3], [0.2,0.3]]. (See shape for reference.)

[0110] At box 212, for each video frame, object metadata is detected and used as supplementary input. For example, the obtained sample object labels could be [Phone: [100,100], Human_0: [200,200], Human_2: [900,900]]. For such an example, for [Phone: [100,100]], "Phone" indicates the object category, and [100,100] could indicate the object's location within the frame. It should be understood that the above description is simplified for comprehension purposes only. For example, during implementation, the object category could be indicated by a categorical integer, and further, the value labeled for that object category could be 4-dimensional data and / or normalized using the frame size.

[0111] At box 214, the Region of Interest Align (RoIAlign) process is performed to obtain human and object regions for cropping. At box 216, a set of human and object tokens is obtained. In this example, objects not in the RoI are removed. The example object labels obtained at box 216 could be [Phone: [100, 100], Human_0: [200, 200]]. Therefore, Human_2 has been removed.

[0112] At box 218, the first cross-attention mechanism is performed using the extracted human token and extracted object token from box 216, and the extracted initial image patch token from box 210. Refer to equations (2) and (3). At box 220, a set of processed image patch tokens containing object guidance information is obtained. It should be understood that each processed image patch token containing object guidance information is a multidimensional tensor, such as a 768-dimensional tensor. In one example, there may be 16×16 image patch tokens for each frame. In one example, there may be example image patch tokens of a 3D tensor [0.4, 0.5, 0.1]. After performing the first cross-attention mechanism using the extracted human token and extracted object token, for example, if there are two objects in the example, one of which is located within or contained in the example image patch token, and the other is not contained in the example image patch token, the example image patch token may simply change to [0.6, 1.0, 0.2], [0.04, 0.05, 0.01]. By performing max pooling on axis 0 (object orientation), a processed image block token containing object guidance information is obtained at box 220, with values ​​[0.6, 1.0, 0.2]. It will be understood that each value can be broadly considered to represent / describe one aspect of the token. It should be understood that the above description is simplified for illustrative purposes only, and the values / numbers are for illustrative purposes only.

[0113] At box 228, the group of processed image block tokens containing object guidance information is processed by a classification head including a fully connected layer. Classification head 228 is configured, for example, to use action recognition from OGA branch / module 204 based on feature prediction results. Example predictions derived by classification head 228 from OGA branch / module 204 could be {"playing on the phone": 0.5, "eating": 0.6}.

[0114] For DIA module 206, at box 222, an enumeration process is performed using the human token and corresponding other object tokens O extracted for each video frame from box 216. The enumeration function creates M = m*n rows of HO pairs, and for each pair, the H token is not yet concatenated with the O token. For example, the H token can have [1×j] dimensions, the O token can have [1×k] dimensions, and after the enumeration process, the pair is presented as, for example, [[1×j], [1×k]]. Next, the concatenation process joins the data together in the format [1×(j+k)] for processing in subsequent stages / boxes. During model training, it should be understood that even if the same elements are used, a tensor of shape [1×j, 1×k] and a tensor of shape [1×(j+k)] are different. Among the M rows of HO tokens, there is usually a most decisive token, which has a greater impact on the prediction result than the other tokens. This is understood as how the model can select the appropriate HO token, i.e., by assigning the most appropriate (most decisive) token with the highest weight. In this example, during the training phase, the human-phone pair occurs more frequently than other human-object pairs, therefore the human-phone token is assigned a higher weight, making it the most decisive token in this case among the M rows (tokens) created by the splicing process at box 222. In this example, there could be an object token for a baby car seat. An example of HO token splicing could be [human_0-phone: [0.24, 0.44], human_0-baby seat: [0.10, 0.83]]. In this example, the [human_0-phone: [0.24, 0.44]] token is the most decisive token among the two tokens shown. It will be understood that each value of the HO token can be generally considered to represent / describe one aspect of the HO token. It should be understood that the above description is simplified for comprehension purposes only, and the values / numbers are for illustration only.

[0115] At box 224, Pos data or spatial embeddings are provided to the HO tokens to construct a set of HO tokens. Each of the constructed HO tokens now includes information associated with visual relationships (stitching) and information (containing Pos data) related to the spatial relationships between human tokens and object tokens. (See reference.) Figure 3 An example of an aggregation of HO tokens with location information or Pos data could be [[0.13, 0.43, 200, 200], [0.13, 0.45, 400, 400]]. It should be understood that the values ​​of HO tokens embedded with Pos data are merely illustrative numerical examples. It should be understood that in one implementation, such HO tokens could each be 768-dimensional tensors. It should be understood that the above description is simplified for illustrative purposes only.

[0116] At box 226, a second cross-attention mechanism is performed using the constructed HO token from box 224 and the processed image block token containing object guidance information from box 220. At box 230, a set of resulting processed HO tokens is obtained. An example of the resulting processed HO tokens after the second cross-attention mechanism could be [[0.6,0.7], [0.20,0.33], [0.04,0.03]].

[0117] At box 232, a feature aggregation process is performed based on the resulting processed HO token. For example, refer to equations (4) and (5). Equations (4) and (5) describe how the aggregation process is performed / works. As an example, given token ho1 = [1 2 34] 768 (Total 768 values / digits), token ho1 is processed by equation (5) to obtain its weight, for example, a1 = 0.2. Similarly, for another token ho2, its weight can be provided as a2 = 0.8, where ho2 = [2 4 68]. 768 Furthermore, this process can calculate the weights a3, a4, ..., a of rows M. M The feature aggregation process continues to process / compute intermediate features z = a1*ho1 + a2*ho2 + ... + a by assigning weights to each token in the processed HO tokens obtained from this group. m *ho m(See equation (4)). In this process, equation (5) provides the largest number, such as a2, and then it can be seen that ho2 is the most decisive HO token. The result provided by the backpropagation of model training is that the parameters of equation (5) can eventually provide the correct / appropriate weights for each token. In an example simplified implementation of the feature aggregation process, a set of random parameters can be first passed to equation (5) and processed using that equation. These parameters are used to infer the model and provide, for example, predictions processed by equations (5) and (4). If the predictions are not appropriate, this set of parameters can be changed based on the degree of inappropriateness of the predictions, and another set of parameters can be iteratively processed by the equation. Thus, feature aggregation module 232 is arranged to use human-object (HO) interactions from DIA branch / module 206 to predict the results. An example prediction from DIA branch / module 206 by feature aggregation module 232 could be {"playing on the phone": 0.95, "eating": 0.2}. As described above, the different classifications of action-level features (or labels) from 1 to N (or, for example, nominally represented as 0 to N-1) are schematically shown at number 238 in feature aggregation module 232. For the example above, {“playing on the phone”: 0.95, “eating”: 0.2} indicates that the prediction provided by the DIA branch is that “playing on the phone” has a probability of 0.95 and “eating” has a probability of 0.2, and the number of predicted action categories C is predetermined / fixed, depending on the training dataset. In this example, C is 2, referring to the two possible actions mentioned above. That is, in this simplified example, the model makes a prediction between these two action categories. It should be understood that the above description is simplified for comprehension purposes only.

[0118] At box 234, a prediction module 234 is provided to receive prediction results from classification head 228 and feature aggregation module 232 to provide an action recognition prediction result. Prediction module 234 is arranged to perform a weighted summation process to obtain the prediction result. An example of the weighted summation calculation / process could be "playing on the phone" probability = [0.1 x probability_OGA_branch] + [0.9 x probability_DIA_branch]. The "playing on the phone" action is predicted by classification head 228 and feature aggregation module 232 (i.e., probability_OGA_branch and probability_DIA_branch, respectively). The weights 0.1 and 0.9 can be predetermined and are described here only as an example. In this example, an example output of prediction module 234 (i.e., after fusing the two branches OGA branch / module 204 and DIA branch / module 206) could be {"playing on the phone": 0.9, "eating": 0.1}. This output is transmitted from prediction module 234 as an action recognition prediction result. Based on the action recognition prediction results, for example, when the summation weight of the action classification is 1, the most likely action can be determined. For example, in this case, based on weights of 0.9 and 0.1, the most likely action is "playing on the phone".

[0119] In the following description, exemplary implementations of ORA module 202 will be described using the following technical details.

[0120] For the dataset, the Drive&Act dataset taught in the following literature is used: M. Martin et al., Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles, in: Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019, pp. 2801-2810. The specific content of M. Martin et al. is incorporated herein by reference. The Drive&Act dataset is widely recognized for its extensive use in automotive cockpit-related tasks, including action recognition and multimodal learning. This dataset consists of 12 coarse-grained scenes and 34 fine-grained action classifications, with five viewpoints (front-top, top-right, rear, frontal view, top-left) and three different modalities (RGB (visible), infrared, and depth). In this exemplary implementation, a front-top view with fine-grained annotations in the NIR (near-infrared) mode is used as the motion recognition setting to obtain a wider field of view and light-invariant features.

[0121] For training ORA model 202, a pre-trained VideoMAE base model was used as the backbone network. For input, the raw video or video data was truncated into relatively small video segments, each 90 video frames long, based on the video category during preprocessing. The smaller processed video segments were temporally sampled to randomly collect 16 video frames. The video segments were resized to 224×224 and then subjected to a RandomFlip operation, for example, available in PyTroch. The token embedding size was determined to be (2, 16, 16), and the feature dimension was determined to be 768, aligned with VideoMAE taught by Z. Tong et al.

[0122] The inventors recognized that since the Drive&Act dataset does not provide bounding box annotations for object detection, for the example implementation, an off-the-shelf detector trained on a manually annotated subset was used to obtain the detected objects for the entire dataset. More than 1500 infrared images were extracted from the raw video or video data, and more than 8200 object instances were extracted and identified. Fifteen categories of objects appearing in the videos were labeled, including but not limited to people, wearable objects (glasses, jackets, etc.), interactive objects (bottles, food, cigarettes, mobile phones, etc.), and other common objects in the automotive cabin environment (seat belts, backpacks, bags, laptops, paper, etc.).

[0123] As described above, in this exemplary embodiment, an off-the-shelf object detector (such as PP-YOLOE, as taught in the following literature: S. Xu et al., Pp-yoloe: An evolved version of yolo, arXiv preprint arXiv:2203.16250 (2022)) is used to detect all objects in the raw video or video data. The specific content of S. Xu et al. is incorporated herein by reference. In this exemplary embodiment, OGA and DIA branches are introduced in the last three layers (10, 11, 12) of the ViT VideoMAE backbone network (corresponding to OGA module 204, DIA module 206, and feature aggregation module 232). The maximum number of objects M in a frame is set to 4 (for example, according to the parameters recommended in the following literature (multiple): R. Herzig et al., Object-region video transformers, in: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 3148-3159 (the contents of which are incorporated herein by reference)), and the maximum HO logarithm N is correspondingly determined to be 6. The weight λ of the HO branch is empirically set to 0.2.

[0124] In this exemplary implementation, all experiments were conducted using four RTX 3090 graphics processing units (GPUs) with a batch size of 8. The AdamW optimizer (a stochastic optimization method / process) was used with segmented learning rates of [10e-4, 10e-5, 10e-6] over [0, 20, 30] epochs, for example, according to the parameters(s) recommended by R. Herzig et al. as described above. In this exemplary implementation, the ORA model was trained for a total of 35 epochs using label-smooth cross-entropy loss.

[0125] Then, the exemplary implementation of ORA Module 202 is evaluated using the Driver Monitoring Dataset Drive&Act and fairly compared with other known action recognition models.

[0126] Following the research of M. Martin et al. and the literature of D. Lin et al. ("Multi-modality action recognition based on dual feature shift in vehicle cabinmonitoring", arXiv preprint arXiv:2401.14838 (2024) – the contents of which are incorporated herein by reference), the inventors set Top-1 accuracy and balanced accuracy as the primary evaluation metrics. Balanced accuracy is the average of the Top-1 accuracy for each class, which better reflects the model's ability to handle imbalanced classification. Since the Drive&Act dataset provides three different dataset splits, the results are the average of the three splits performed according to the previous method. Table 1 below lists the results of the state-of-the-art action recognition method and the ORA model (shown in the last row of Table 1).

[0127] method Top-1 Acc. Top-5 Acc. Bal.Acc. UniformerV2 70.91 93.15 59.57 ViT-Base 79.60 97.50 61.59 ORViT 82.62 96.59 65.62 ORA 85.49 98.31 68.96

[0128] Table 1

[0129] As shown in Table 1, the ORA model has a Top-1 Acc. value of 85.49 and a Bal.Acc. value of 68.96, which are better than other ViT backbone networks and existing models. Therefore, it can be seen that the ORA model outperforms other state-of-the-art methods.

[0130] In analyzing one or more exemplary ORA models, the inventors recognized that the ORA model differs from some prior art research in several aspects. For example, the ORA model is based on a Transformer-based model, which may be fundamentally different from other forms of machine learning models such as CNNs. Attention mechanisms can provide greater robustness potential in dynamic environments. For example, the ORA model utilizes appearance-based feature detection. This can effectively enable learning models that understand video features. For example, processed image patch tokens are used together with HO tokens for mid-stage fusion (e.g., before feature aggregation, rather than late-stage fusion in some studies), compared to, for example, numbers 216, 218, 220, 222, and 224, which result in, for example, the second cross-attention mechanism at number 226. For example, the ORA model utilizes object detection, which can effectively provide the ORA model with location information and provide object categories / classifications for additional features. In one or more exemplary embodiments, object labels and locations are passed to the ORA model, and this can allow additional features of human-object cross-reference information to be passed to the action classifier. For example, the ORA model of one or more exemplary embodiments does not require additional word embedding processing on the image / frame and / or does not require adding pose information to the image / frame. Because this additional input is not required, the flexibility of using the ORA model in more applications is improved. Furthermore, the exemplary embodiments are geared towards action recognition, which is broadly used for the recognition and classification of human actions in videos. Action recognition is the opposite of HOI tasks, which are typically used to identify a set of interactions in an image. HOI tasks typically involve the localization of interacting subjects (e.g., humans) and targets (i.e., objects), as well as the classification of interaction labels. It can be recognized that in the exemplary embodiments of action recognition, object information is used to enhance the accuracy of action recognition, such as human actions, while the interaction between the subject and the target is largely unused or irrelevant to such exemplary embodiments.

[0131] Figure 4A Figure 4 is a schematic diagram illustrating a typical Transformer model structure. It will be understood that the exemplary embodiments described herein utilize a transformer-based model structure. Figure 4 is provided to introduce some concepts associated with the transformer-based model structure, which are familiar to those skilled in the art.

[0132] The transformer architecture is a deep learning model architecture that includes several feature components. The transformer allows the use of self-attention mechanisms, where attention allows the model to focus on different parts of the input sequence when making predictions. In a transformer, each token in the input can pay attention to every other word to capture dependencies, regardless of distance within the sequence. Self-attention refers to the process by which a token pays attention to other tokens in the same sequence to understand the context. The transformer also utilizes positional encodings added to the input embedding to provide the model with information about the relative positions of tokens in the sequence. Furthermore, the transformer can use multiple attention mechanisms in parallel (multi-head). This allows the model to capture different types of relationships between tokens. Additionally, the transformer can contain a feedforward neural network with fully connected layers to further process the output from the attention layers. The transformer can have normalization layers and residual connections after each layer, which helps in training deeper networks. Furthermore, the transformer can include encoder-decoder structures, such as those shown in numbers 402 and 404, respectively. For the encoder (e.g., 402), the encoder processes the input sequence and produces a set of codes. Each code represents the input with contextual information from the entire sequence. For a decoder (e.g., 404), the decoder uses encoding to generate an output sequence (e.g., in a different format).

[0133] Transformers offer several advantages, including parallelization (as opposed to sequential processing), as they can process the entire sequence at once, enabling faster training through parallel processing; and the ability to handle long-range dependencies, as the self-attention mechanism allows transformers to effectively capture long-range dependencies in a sequence.

[0134] Since video can be viewed as a sequence of frames (images), the transformer architecture is applicable to video processing. Therefore, the transformer can process the sequence of frames in a video. Each frame can be processed in the same way that words or tokens are processed in natural language processing. Recognizing that both spatial (within a frame) and temporal (across frames) information can be important in video processing, the transformer can be tuned to capture this information through a spatiotemporal attention mechanism. This mechanism allows the model to focus on relevant parts of the video across both time and space. Building on these advancements, the Visual Transformer (ViT) can be extended to video by processing video frames as sequences of image patches. Typically, each frame is divided into multiple image patches, and these image patches across multiple frames can be fed into the transformer to learn spatial and temporal dependencies. During partitioning, each frame in the video can be divided into non-overlapping or overlapping image patches. Such image patches are typically square regions of the frame, such as 16×16 or 32×32 pixels. Typically, each image patch is flattened into a 1D vector and then passed through a linear projection layer (e.g., a fully connected layer) to create a low-dimensional embedding. This process converts each image patch into a token that can be processed by the transformer. Further, to help the transformer understand the order or location of the image patches, positional encodings are added to the image patch tokens. These encodings allow the model to understand spatial locations within a frame and, when processing video, to understand the temporal order across frames. In some applications, the transformer architecture can be extended to operate in three dimensions, where the third dimension represents the temporal aspect. Such a model can learn features directly from the spatiotemporal volume of video data. In some applications, the transformer can use a hierarchical approach, where it operates at multiple scales, for example, processing short sequences of frames first (e.g., capturing fine details) and then processing longer sequences (e.g., capturing broader context).

[0135] In the exemplary embodiment, cross-attention has been utilized. In the context of transformers, cross-attention is a mechanism that allows one sequence of data to pay attention to another sequence, facilitating the integration of information across different data sources or stages in the model. Cross-attention is typically used in the decoder part of a transformer architecture, where the model attempts to generate an output sequence based on the input sequence. Typically, for cross-attention, the attention mechanism computes a weighted sum of the input elements, where the weights (attention scores) indicate how much attention each input element should receive. This allows the model to consider relevant parts of the input when generating the output. A transformer can utilize self-attention, where each element in a sequence pays attention to other elements in the same sequence to capture internal relationships. A transformer can utilize cross-attention, where an element in one sequence (e.g., the current output sequence of the decoder) pays attention to an element in another sequence (e.g., the output of the encoder). This allows the model to incorporate external information from the encoder when generating the output sequence. Generally, the process of cross-attention involves a query from the current layer of the decoder (e.g., related to the generated output sequence); and a key and value from the encoder output (e.g., related to the input sequence). Typically, decoders use cross-attention to determine which parts of the encoder output (e.g., representations of the input sequence) are most relevant to the current decoding step. This helps the model produce context-appropriate output. It should be recognized that cross-attention can be used in models that integrate information from different modalities (e.g., text and images in visual question answering, or spatial and visual relationships), allowing text to pay attention to image features and vice versa. In summary, cross-attention can be a powerful mechanism in transformer architectures that facilitates interaction between different sequences, enabling the model to generate outputs with in-depth understanding of relevant parts of the input sequence. Cross-attention can be used in tasks that leverage the close relationship between input and output.

[0136] In an exemplary embodiment, a classification head is described. In the context of a transformer, a classification head is a component typically located on top of the core transformer architecture and used to perform a final task-specific prediction (typically for classification tasks). The classification head transforms the transformer's output into a format suitable for making a decision (such as predicting class labels). A classification head is typically a small neural network that may include a feedforward layer or several layers that take the final output from the transformer and transform it into a probability distribution of possible classes. In the case of multi-class classification, the output of the classification head may be a softmax layer, generating a probability for each class.

[0137] In an exemplary embodiment, the RoIAlign technique is described. Generally, RoIAlign is a technique for accurately extracting and aligning features from specific regions of an image, typically proposed by region proposal methods. In object detection models, regions of interest (RoIs) are typically proposed as potential bounding boxes where objects can lie. These RoIs are then mapped onto feature maps. In visual transformer (ViT) adaptation, features from different image patches (or tokens) may need to be accurately aligned with the regions of interest. RoIAlign can be used to ensure that the features extracted from these regions are relatively accurate and preserve spatial information.

[0138] In an exemplary embodiment, feature aggregation is proposed. Generally, feature aggregation refers to the process of combining or pooling features (representations) generated by a transformer to produce a summarized or merged representation that can be used for downstream tasks such as classification, sequence labeling, or other forms of decision-making. A transformer typically generates a sequence of feature vectors (also called embeddings) corresponding to input tokens. Each vector can represent a context-rich input element from the entire sequence, using a self-attention mechanism. For different tasks, such as those requiring a single output (e.g., image classification or action recognition), it is beneficial to aggregate these feature vectors into a single compact representation. This representation can then be fed into a classification head or another model component for final decision-making.

[0139] Figure 4B This is a schematic diagram illustrating an example Transformer model structure in an exemplary embodiment. Example model structure 406 can be used in the exemplary embodiments described herein.

[0140] In the exemplary embodiment, a transformer encoder section (reference numeral 402) is used. A multi-head attention setup 408 is used, as schematically shown. For simplicity, other structures / sections of the ORA module are not shown, and it should be understood that these sections are designed to facilitate feature operations that provide input to the transformer structure 406.

[0141] The inventors recognized that a decoder section was not required in the exemplary embodiment (as a classification task) (see figure 404).

[0142] As an example, refer to Figure 2Boxes 208, 218, and 226 are implemented as encoder boxes. For example, at input module 208, a frame can be provided as input, and image patch features (or image patch tokens) can be obtained as output. This corresponds to box 210 with initial image patch tokens. For example, at box 218, for a first cross-attention mechanism, one or more extracted human tokens and one or more extracted object tokens (see box 216) and the initial image patch token (see box 210) can be provided as input, and a processed image patch token containing object guidance information (see box 220) can be obtained as output. For example, at box 226, for a second cross-attention mechanism, a constructed HO token (see box 224) and a processed image patch token containing object guidance information (see box 220) can be provided as input, and the resulting processed HO token can be obtained as output (see box 230).

[0143] Figure 5 This is a schematic block diagram 500 illustrating a method for object-enhanced action recognition in an exemplary embodiment. The method is computer-implemented. In step 502, one or more initial image patch tokens are obtained from one or more video frames. In step 504, the one or more initial image patch tokens are processed to extract one or more human tokens and one or more object tokens. In step 506, the extracted one or more human tokens, the extracted one or more object tokens, and the one or more initial image patch tokens are applied to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object guidance information. In step 508, the extracted one or more human tokens and the extracted one or more object tokens are enumerated to construct a set of human-object (HO) tokens, each of which includes information associated with a visual relationship and information associated with a spatial relationship between the human token and the object token. In step 510, the constructed HO tokens and the processed image patch tokens containing object guidance information are applied to a second cross-attention mechanism to obtain a set of resulting processed HO tokens. In step 512, a feature aggregation process is applied to the resulting processed HO tokens to synthesize the features of M sets of resulting processed HO tokens into a single feature representation. In step 514, a prediction result is obtained based on the set of processed image block tokens containing object guidance information. In step 516, an action recognition prediction result is obtained based on the result of the feature aggregation process and the prediction result based on the set of processed image block tokens containing object guidance information. In step 518, the action recognition prediction result is output for use.

[0144] In an exemplary embodiment, the step of processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens may include using object metadata information to extract the one or more human tokens and one or more object tokens, the object metadata information being based on applying bounding box coordinates and object categories from object annotation data to the one or more video frames. The step of processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens may further include sorting the extracted one or more human tokens and one or more object tokens to place the one or more human tokens in front of the one or more object tokens. The step of enumerating the extracted one or more human tokens and one or more object tokens to construct a set of HO tokens may further include embedding information associated with visual relationships using a stitching process and a positional encoding process. The step of enumerating the extracted one or more human tokens and one or more object tokens to construct a set of HO tokens may further include embedding information associated with spatial relationships using spatial embedding data and another stitching process. The step of applying a feature aggregation process may include combining the different obtained processed HO tokens into M groups, where each group represents an action-level feature trained using action annotations; and aggregating the obtained processed HO tokens using weights applied to each obtained processed HO token. The step of applying the feature aggregation process to the resulting processed HO token may further include selecting the most decisive processed HO token for action recognition. The step of outputting the action recognition prediction result for use may include: transmitting the action recognition prediction result to an action device, and causing the action device to activate an action based on the action recognition prediction result.

[0145] In an exemplary embodiment, a non-transitory tangible computer-readable storage medium may be provided, having stored thereon software instructions that, when executed by a computer processor of an object augmentation action recognition system, cause the computer processor to perform a computer-implemented method of object augmentation action recognition by executing the following steps, as described above, for example, with reference to... Figure 5 As stated above.

[0146] In an exemplary embodiment, object tokens (including human tokens) are generated using object annotation information. These object tokens, along with initial image patch tokens obtained from video frames, are then applied to a first cross-attention mechanism. One technical effect / benefit is that this mechanism allows for targeted consideration of object-related information and further ensures that the model's attention can be fine-tuned to object-related details.

[0147] In an exemplary embodiment, object annotation is also used to distinguish between humans and objects, and to generate human-object (HO) tokens that encapsulate the visual features and spatial relationships of object tokens (including human tokens). These HO tokens are then applied together with processed image patch tokens to a second cross-attention process. One technical effect / benefit is that this mechanism allows for the perception and understanding of holistic / global actions.

[0148] In an exemplary embodiment, one technical effect / benefit of the feature aggregation process is that token-level features can be merged into a unified action-level output. For example, by clustering different HO tokens under action-level annotation supervision, the feature aggregation process can select the most decisive HO token for action recognition.

[0149] One technical problem that this embodiment can solve is how to obtain more accurate action recognition results. A technical solution can utilize the various technical effects / benefits mentioned above to provide a more object-related or object-guided, and relationship-guided (spatial and / or visual relationship between humans and objects) action recognition process, thereby providing more accurate action recognition prediction results.

[0150] In the described exemplary embodiments, an efficient bi-branch action recognition model is provided, which can combine object visual information (or object annotation information) and human-object relationship information. Furthermore, a feature aggregation module is provided to facilitate efficient supervision of relationship-level HO features using action-level annotations. The inventors recognize that the proposed ORA model is applicable to any transformer-based action recognition model.

[0151] In providing exemplary embodiments, the inventors recognized that interactions between people (or humans) and objects are not merely useless information, but can be an indispensable part of accurately understanding human actions. For example, multiple objects may appear simultaneously in a video clip, and the inventors recognized that not all objects are relevant; only a subset of key objects may have a causal relationship with human actions. Therefore, the inventors recognized that understanding interactions can improve the action recognition performance of action recognition models.

[0152] In providing exemplary embodiments, the inventors propose a novel framework called Object-Enhanced Relationship-Aware Action Recognition (ORA), as described in the exemplary embodiments. The ORA model is specifically designed to integrate and leverage dynamic interactions between humans and objects. In the exemplary embodiments, the ORA model employs a two-branch approach, which includes, for example, an Object-Guided Attention (OGA) module and a Dynamic Interaction Attention (DIA) module. The OGA module can utilize object annotation information to generate object tokens, which are then used together with image patch tokens obtained from video frames to participate in a first cross-attention mechanism. This mechanism effectively allows for targeted consideration of object-related information and may guide the ORA model's attention to be more finely tuned to object-related details / information / data. Simultaneously, the DIA module can use object annotations to distinguish between humans and objects, and can then generate / create human-object (HO) tokens that encapsulate / include visual features and spatial relationships of the object tokens. That is, these human-object (HO) tokens are based on the visual and spatial relationships of the object tokens. Then, such HO tokens are used together with the processed image block tokens from the OGA module for the second cross-attention process / mechanism, thereby effectively improving / enhancing the DIA module's ability to perceive and understand the overall motion of the video segment, i.e., enhancing overall motion perception.

[0153] Furthermore, in providing exemplary embodiments, the inventors recognized a potential inconsistency between the scale of the token-level features and the scale of the desired action-level output. The inventors proposed a feature aggregation module that utilizes attention mechanisms and gating methods to add weights and merge token-level features into a consistent action-level output, e.g., a unified action-level output. This can usefully and effectively address the inconsistency problem and ensure a relatively harmonious / consistent feature representation. In other words, it can resolve the misalignment between the output scale and the feature granularity.

[0154] In one or more of the exemplary embodiments described, the effectiveness of the ORA model has been rigorously evaluated on the Drive&Act dataset, considered a challenging benchmark for action recognition. Experimental results demonstrate significant performance improvements compared to baseline models and other action recognition frameworks. These results validate the effectiveness of the ORA model and highlight the important consideration of incorporating detailed object and human interaction dynamics to achieve more accurate and context-rich action recognition.

[0155] Figure 6 This is a schematic diagram of a computer system suitable for implementing exemplary embodiments.

[0156] Different exemplary embodiments can be implemented within the context of data structures, program modules, programs, and computer instructions that execute in a computer-implemented environment. This document briefly discloses a general-purpose computing environment. One or more exemplary embodiments can be implemented in one or more computer systems, such as... Figure 6 This is shown schematically.

[0157] One or more exemplary embodiments may be implemented as software, such as a computer program that executes within computer system 600, and instruct computer system 600 to perform or implement methods of exemplary embodiments.

[0158] Computer system 600 includes computer unit 602, input modules (such as keyboard 604 and pointing device 606), and multiple output devices (such as display 608 and printer 610). Users can interact with computer unit 602 using these devices. Pointing can be implemented using a mouse, trackball, pen device, or any similar device. One or more other input devices (not shown), such as joysticks, game controllers, satellite antennas, scanners, touchscreens, etc., can also be connected to computer unit 602. Display 608 may include cathode ray tube (CRT), liquid crystal display (LCD), field emission display (FED), plasma display, or any other device that produces images viewable by the user.

[0159] Computer unit 602 can be connected to computer network 612 via a suitable transceiver device 614 to access, for example, the Internet or other network systems such as a local area network (LAN), wide area network (WAN), or personal network. Network 612 may include servers, routers, network personal computers, peer-to-peer devices or other common network nodes, cordless phones, or wireless personal digital assistants. The networking environment can exist in offices, enterprise-wide computer networks, and home computer systems, etc. Transceiver device 614 can be a modem / router unit located inside or outside computer unit 602, and can be any type of modem / router, such as a cable modem or satellite modem.

[0160] It should be understood that the network connection shown is exemplary, and other methods for establishing communication links between computers can be used. It is assumed that various protocols exist, such as TCP / IP, Frame Relay, Ethernet, FTP, HTTP, etc., and that computer unit 602 can operate in a client-server configuration to allow a user to retrieve web pages from a web-based server. Furthermore, various web browsers can be used to display and manipulate data on the web pages.

[0161] The computer unit 602 in the example includes a processing unit or processor 618, random access memory (RAM) 620, and read-only memory (ROM) 622. ROM 622 may be system memory storing basic input / output system (BIOS) information. RAM 620 may store one or more program modules, such as operating system, application programs, and program data.

[0162] Computer unit 602 further includes multiple input / output (I / O) interface units, such as I / O interface unit 624 to display 608 and I / O interface unit 626 to keyboard 604. Components of computer unit 602 are typically communicated and interfaced / coupled via interconnect system bus 628 in a manner known to those skilled in the art. Bus 628 can be any of several types of bus architectures, including memory bus or memory controller, peripheral bus, and local bus using any of various bus architectures.

[0163] It should be understood that other devices can also be connected to system bus 628. For example, a video or digital camera can be coupled to system bus 628 using a Universal Serial Bus (USB) interface. An additional device can be coupled to computer unit 602 using an IEEE 1394 interface. Other manufacturer interfaces (such as FireWire developed by Apple and i.Link developed by Sony) are also possible. Device coupling to system bus 628 can also be achieved through parallel ports, gaming ports, PCI boards, or any other interface used to couple input devices to the computer. It should also be understood that, although components are not shown in the figure, microphones and speakers can be used to record and reproduce sound / audio. A sound card can be used to couple microphones and speakers to system bus 628. It should be understood that several peripheral devices can be coupled to system bus 628 simultaneously through alternative interfaces.

[0164] An application can be provided to a user of computer system 600, encoded / stored on a data storage medium such as a CD-ROM or flash memory. The application can be read using a corresponding data storage medium drive of data storage device 630. The data storage medium is not limited to portable media, but may also include those embedded in computer unit 602. Data storage device 630 may include a hard disk interface unit and / or a removable memory interface unit (neither shown in detail), coupling the hard disk drive and / or removable memory drive to system bus 628, respectively. This enables data reading / writing. Examples of removable memory drives include disk drives and optical disk drives. The drive and its associated computer-readable medium (such as a floppy disk) provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer unit 602. It should be understood that computer unit 602 may include several such drives. Furthermore, computer unit 602 may include drives for interface connection with other types of computer-readable media.

[0165] The application is read by processor 618 and its execution is controlled. Intermediate storage of program data can be implemented using RAM 620. The methods(s) of the exemplary embodiments can be implemented as computer-readable instructions, computer-executable components, or software modules. Alternatively, one or more software modules can be used. These modules may include executable programs, data link libraries, configuration files, databases, graphics, binary data files, text data files, object files, source code files, etc. When one or more computer processors execute one or more software modules, the software modules interact to cause one or more computer systems to perform in accordance with the teachings herein.

[0166] The operation of computer unit 602 can be controlled by various program modules. Examples of program modules include routines, programs, objects, components, data structures, libraries, etc., that perform specific tasks or implement specific abstract data types. Exemplary embodiments can also be implemented using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, personal digital assistants, mobile phones, etc. Furthermore, exemplary embodiments can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked via wireless or wired communication networks. In a distributed computing environment, program modules can reside on both local and remote storage devices.

[0167] In the description herein, unless otherwise stated, the terms “coupled” or “connected” are intended to cover direct connections or connections via one or more intermediate means. The terms “adjacent,” “previous,” and “next” are intended to cover direct or indirect adjacency, or connections between two layers / parts / components / objects with one or more intermediate layers, etc.

[0168] In the description herein, the terms "first," "second," "third," "fourth," etc., are used to refer to parts / components of the corresponding figures. These terms are not intended to limit or restrict to any particular part / component. Rather, they should be understood in conjunction with the corresponding numbers and their assigned meanings.

[0169] Unless otherwise specified, “a / an” or “the” is intended to mean “one or more”.

[0170] The terms “configured to (perform a task / action)”, “configured for (perform a task / action)”, etc., as used in this specification include programmable, programmed, connectable, wired, or otherwise constructed to have the ability to perform a task / action when arranged or installed as described herein. The terms “configured to (perform a task / action)”, “configured for (perform a task / action)”, etc., are intended to cover “performing a task / action when in use,” such as being specifically used for and / or specifically configured and / or specifically arranged and / or specifically suited for performing or executing a task / action.

[0171] The term “and / or,” such as “X and / or Y,” should be understood to mean “X and Y” or “X or Y,” and should be understood to provide explicit support for both or either meaning. Unless otherwise specified, the use of “or” is intended to indicate “inclusive or,” not “exclusive or.”

[0172] The terms "associated with" or "related to" as used in this article refer to a broad relationship between two elements. This relationship includes, but is not limited to, physical, chemical, or biological relationships. For example, when element A is associated with element B, elements A and B can be directly or indirectly attached to each other, or element A can contain element B, and vice versa.

[0173] The terms “exemplary embodiment,” “example embodiment,” “exemplary implementation,” “exemplary method,” “exemplary,” etc., used herein are intended to indicate examples of things described in this disclosure. Such examples may relate to one or more features as defined in the claims and are not necessarily intended to emphasize the best example or the necessity of any feature.

[0174] The descriptions herein may be explicitly or implicitly described in certain sections as algorithmic and / or functional operations that manipulate data within computer memory or electronic circuits. These algorithmic and / or functional descriptions are typically used by those skilled in the art of information / data processing for efficient description. Algorithms generally involve a consistent sequence of steps that produce a desired result. Algorithmic steps may include the physical manipulation of physical quantities (e.g., electrical, magnetic, or optical signals that can be stored, transmitted, transferred, combined, compared, and otherwise manipulated).

[0175] Furthermore, unless otherwise specified and as will generally be apparent from the following text, those skilled in the art will understand that throughout this specification, discussions using terms such as “scan,” “calculate,” “determine,” “replace,” “generate,” “initialize,” and “output” refer to the actions and processes of an instruction processor / computer system or similar electronic circuits / devices / components that manipulate / process data represented as physical quantities within the system and convert it into other data similarly represented as physical quantities within the system or other information storage, transmission, or display devices.

[0176] The specification also discloses related devices / apparatus for performing the steps of the described methods. Such devices may be specifically constructed for the purposes of these methods, or may include general-purpose computers / processors or other devices selectively activated or reconfigured by computer programs stored in storage components. The algorithms and displays described herein are not inherently related to any particular computer or other device. It should be understood that general-purpose devices / machines may be used in accordance with the teachings herein. Alternatively, it may be desirable to construct dedicated devices / apparatus to perform the method steps.

[0177] Furthermore, this invention also implicitly encompasses computer programs, as it is apparent that the steps of the methods described herein can be implemented using computer code. It should be understood that various programming languages ​​and codes can be used to implement the teachings described herein. Moreover, where applicable, the computer program is not limited to any particular control flow and different control flows can be used without departing from the scope of this invention.

[0178] Furthermore, if applicable, one or more steps in the steps of a computer program may be performed in parallel and / or sequentially. Such a computer program may be stored on any computer-readable medium, if applicable. Computer-readable media may include storage devices such as disks or optical discs, memory chips, or other storage devices suitable for connection to a suitable reader / general-purpose computer interface. In this case, the computer-readable storage medium is non-transitory. Such storage media also encompasses all computer-readable media, such as media that store data only for short periods and / or only when powered, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may even include wired media, such as those exemplified in Internet systems, or wireless media, such as those exemplified in Bluetooth technology. Computer-readable media may be cloud storage devices, such as those in the Internet or intranets. When the computer program is loaded and executed on a suitable reader, it effectively creates means that can, for example, implement the steps of the described methods in a physical embodiment. The computer-readable medium is intended to be transferable and reproducible because the computer program is reproducible, if applicable.

[0179] Exemplary embodiments can also be implemented as hardware modules. A module is a functional hardware unit designed to be used in conjunction with other components or modules. For example, a module can be implemented using digital or discrete electronic components, or it can be part of an overall electronic circuit, such as an application-specific integrated circuit (ASIC). Those skilled in the art will understand that exemplary embodiments can also be implemented as a combination of hardware and software modules.

[0180] Furthermore, in describing some embodiments, this disclosure may have specified methods and / or processes as a particular order of steps. However, unless otherwise required, it should be understood that the method or process should not be limited to the specific order of steps disclosed. Other orders of steps are also possible. The specific order of steps disclosed herein should not be construed as an undue limitation. Unless otherwise required, the methods and / or processes disclosed herein should not be limited to steps performed in the order written. The order of steps may be changed and remains within the scope of this disclosure.

[0181] Furthermore, in the description herein, whenever the word “substantially” is used, it should be understood to include, but is not limited to, “all” or “completely”. Additionally, whenever terms such as “comprising” or “comprise” are used, they are intended as non-limiting descriptive language, as they broadly include the elements / components described following these terms, as well as other components not explicitly stated. For example, when “comprising” is used, a reference to “one” is also intended to mean “at least one” of that feature. In the appropriate context, terms such as “consisting” or “consist” can be considered as a subset of terms such as “comprising” or “comprise”. Therefore, in the embodiments disclosed herein that use terms such as “comprising” or “comprise”, it should be understood that these embodiments provide instruction for corresponding embodiments using terms such as “consisting” or “consist”. Furthermore, whenever terms such as “about” or “approximately” are used, they generally indicate reasonable changes, such as, but not limited to, a change of ±5% in the disclosed value, or a change of approximately 4% in the disclosed value, or a change of approximately 3% in the disclosed value, or a change of approximately 2% in the disclosed value, or a change of approximately 1% in the disclosed value.

[0182] Furthermore, certain values ​​within a range may be disclosed in the description herein. Values ​​showing the endpoints of a range are intended to illustrate a preferred range. Whenever a range is described, it is intended to cover and teach all possible subranges and the individual values ​​within that range. That is, the endpoints of a range should not be interpreted as inflexible limitations. For example, a range description of 1% to 5% is intended to specifically disclose subranges such as 1% to 2%, 1% to 3%, 1% to 4%, 2% to 3%, etc., as well as individual values ​​within that range, such as 1%, 2%, 3%, 4%, and 5%. It should be understood that the individual values ​​within a range also include integers, fractions, and decimals. Furthermore, whenever a range is described, it is also intended to indicate that the range covers and teaches values ​​up to two additional decimal places or significant figures (where appropriate) from the endpoints of the indicated values. For example, the 1% to 5% range description is intended to specifically disclose ranges of 1.00% to 5.00% and 1.0% to 5.0%, as well as all intermediate values ​​across these ranges (e.g., 1.01%, 1.02%, ..., 4.98%, 4.99%, 5.00% and 1.1%, 1.2%, ..., 4.8%, 4.9%, 5.0%, etc.). The intent of this specific disclosure applies to any range of depth / breadth.

[0183] In the described exemplary embodiments, the term "object" is used to refer to an object in a frame. These objects may include humans or inanimate objects. Therefore, references to objects other than humans may be interpreted as referring to inanimate objects.

[0184] In the described exemplary embodiments, the term "human" is used to refer to the subject of the action recognition task. It should be understood that the exemplary embodiments can potentially be extended to other living / animalistic subjects. In such a potential extension, the term "human" can be understood to encompass such other living / animalistic subjects.

[0185] In the exemplary embodiments described, it should be understood that for action recognition tasks, more than one run or iteration can be performed through the ORA module or the object-enhanced action recognition system. That is, the exemplary embodiments are not limited to a single run.

[0186] In the exemplary embodiments described, it should be understood that the exemplary embodiments are not limited to the described VideoMAE implementation and can be extended to work with any other suitable vision-based transformer architecture.

[0187] In the exemplary embodiments described, it should be understood that the ORA module and the object augmentation action recognition system can be pre-trained before being deployed to any application.

[0188] Those skilled in the art will understand that other variations and / or modifications can be made to specific embodiments without departing from the scope of the invention as broadly described herein. For example, features of different exemplary embodiments may be mixed, combined, interchanged, merged, adopted, modified, included, etc., among different exemplary embodiments in the description herein. For example, exemplary embodiments are not necessarily mutually exclusive, as some embodiments may be combined with one or more embodiments to form new exemplary embodiments. Furthermore, it should be understood that although this disclosure provides embodiments having one or more features / characteristics discussed herein, one or more of these features / characteristics may also be excluded in other alternative embodiments, and this disclosure supports such exclusions and these related alternative embodiments. Therefore, the present embodiments are to be considered illustrative rather than restrictive in all respects.

Claims

1. A computer-implemented method of object-augmented action recognition, the method comprising: obtaining one or more initial image patch tokens from one or more video frames; processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens; applying the extracted one or more human tokens and the extracted one or more object tokens and the one or more initial image patch tokens to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object-guided information; enumerating the extracted one or more human tokens and the extracted one or more object tokens to construct a set of human-object (HO) tokens, the constructed HO tokens each including information associated with a visual relationship and information associated with a spatial relationship between a human token and an object token; applying the constructed HO tokens and the processed image patch tokens containing object-guided information to a second cross-attention mechanism to obtain a set of resulting processed HO tokens; applying a feature aggregation procedure to the resulting processed HO tokens to aggregate features of M groups of the resulting processed HO tokens into a single feature representation; obtaining a prediction result based on the set of processed image patch tokens containing object-guided information; obtaining an action recognition prediction result based on a result of the feature aggregation procedure and based on the prediction result of the set of processed image patch tokens containing object-guided information; and outputting the action recognition prediction result for use. The step of processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens includes using object metadata information to extract the one or more human tokens and the one or more object tokens, the object metadata information based on applying bounding box coordinates and object classes from object annotation data to the one or more video frames.

2. The computer-implemented method of claim 1, wherein, The step of processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens further includes ordering the extracted one or more human tokens and the extracted one or more object tokens to place the one or more human tokens in front of the one or more object tokens.

3. The computer-implemented method of claim 1 or 2, wherein, The step of enumerating the extracted one or more human tokens and the extracted one or more object tokens to construct a set of HO tokens further includes using a concatenation procedure and a positional encoding procedure to embed information associated with a visual relationship.

4. The computer-implemented method of any one of claims 1 to 3, wherein, The step of enumerating the extracted one or more human tokens and the extracted one or more object tokens to construct a set of HO tokens further includes using spatial embedding data and another concatenation procedure to embed information associated with a spatial relationship.

5. The computer-implemented method of any one of claims 1 to 4, wherein, The step of applying a feature aggregation procedure includes combining different resulting processed HO tokens into M groups, wherein each group represents action-level features trained using action annotation; and 6. The computer-implemented method of any one of claims 1 to 5, wherein, using weights applied to each resulting processed HO token to aggregate the resulting processed HO tokens. The step of applying a feature aggregation procedure to the resulting processed HO tokens further includes selecting the most determinative resulting processed HO tokens for action recognition.

7. The computer-implemented method of any one of claims 1 to 6, wherein, ​ 8. The computer-implemented method of any one of claims 1 to 7, wherein, The step of outputting the action recognition prediction result for use comprises: transmitting the action recognition prediction result to an action device; and The action device is caused to act based on the action recognition prediction result.

9. An object-augmented action recognition system, the system comprising, an input module for obtaining one or more initial image patch tokens from one or more video frames; an object-augmented relational perception action recognition (ORA) module coupled to the input module, the ORA module comprising: an object-guided attention (OGA) module for processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens, the OGA module being further configured to apply the extracted one or more human tokens and the extracted one or more object tokens and the one or more initial image patch tokens to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object-guided information; a dynamic interaction attention (DIA) module coupled to the OGA module, the DIA module being configured to enumerate the extracted one or more human tokens and the extracted one or more object tokens to construct a set of human-object (HO) tokens, each of the constructed HO tokens comprising information associated with a visual relationship and information associated with a spatial relationship between a human token and an object token, the DIA module being further configured to apply the constructed HO tokens and the processed image patch tokens containing object-guided information to a second cross-attention mechanism to obtain a set of resulting processed HO tokens; a feature aggregation module for applying a feature aggregation procedure to the resulting processed HO tokens to aggregate features of the M sets of resulting processed HO tokens into a single feature representation; a classification head for obtaining a prediction result based on the set of processed image patch tokens containing object-guided information; a prediction module for obtaining an action recognition prediction result based on a result of the feature aggregation procedure and based on the prediction result of the set of processed image patch tokens containing object-guided information; and the prediction module being further configured to output the action recognition prediction result for use; an action device coupled to the ORA module, the action device being arranged to act based on the action recognition prediction result.

10. The object-augmented action recognition system of claim 9, further comprising the OGA module being further configured to extract the one or more human tokens and the one or more object tokens using object metadata information, the object metadata information being based on the OGA module being arranged to apply bounding box coordinates and object classes from object annotation data to the one or more video frames.

11. The object-augmented action recognition system of claim 9 or 10, further comprising the OGA module being further configured to rank the extracted one or more human tokens and the extracted one or more object tokens to place the one or more human tokens in front of the one or more object tokens.

12. The object-augmented action recognition system of any one of claims 9 to 11, further comprising the DIA module to embed information associated with visual relationships using a concatenation process and a position encoding process.

13. The object-augmented action recognition system of any one of claims 9 to 12, further comprising the DIA module to embed information associated with spatial relationships using spatial embedding data and another concatenation process.

14. The object augmented action recognition system of any one of claims 9 to 13, further comprising the feature aggregation module to combine different resulting processed HO tokens into M groups, wherein, Each group represents action-level features trained using action annotations; and to aggregate the resulting processed HO tokens using weights applied to each resulting processed HO token.

15. The object-augmented action recognition system of any one of claims 9 to 14, further comprising the feature aggregation module to perform selecting the most decisive resulting processed HO tokens for action recognition.

16. The object-augmented action recognition system of any one of claims 9 to 15, further comprising the prediction module to transmit the action recognition prediction result to the action device; and to cause the action device to activate the action based on the action recognition prediction result.

17. A non-transitory, tangible computer-readable storage medium having software instructions stored thereon that, when executed by a computer processor of an object-augmented action recognition system, cause the computer processor to perform a computer-implemented method of object-augmented action recognition by performing the following steps, the operations comprising: obtaining one or more initial image patch tokens from one or more video frames; processing the one or more initial image patch tokens to extract one or more human tokens and one or more object tokens; applying the extracted one or more human tokens and the extracted one or more object tokens and the one or more initial image patch tokens to a first cross-attention mechanism to obtain a set of processed image patch tokens containing object guidance information; enumerating the extracted one or more human tokens and the extracted one or more object tokens to construct a set of human-object (HO) tokens, the constructed HO tokens each including information associated with visual relationships and information associated with spatial relationships between human tokens and object tokens; applying the constructed HO tokens and the processed image patch tokens containing object guidance information to a second cross-attention mechanism to obtain a set of resulting processed HO tokens; applying a feature aggregation process to the resulting processed HO tokens to aggregate features of the M groups of resulting processed HO tokens into a single feature representation; obtaining a prediction result based on the set of processed image patch tokens containing object guidance information; obtaining an action recognition prediction result based on a result of the feature aggregation process and based on the prediction result of the set of processed image patch tokens containing object guidance information; and outputting the action recognition prediction result for use. The computer-implemented method of object-augmented action recognition is as claimed in any one of claims 2 to 8.

18. The non-transitory, tangible computer-readable storage medium of claim 17, wherein, ​

Images