Action-Object Recognition in Cluttered Video Scenes Using Text

Abstract

A mechanism is provided for implementing an action-object interaction detection mechanism for recognizing actions in cluttered video scenes. An object bounding box is calculated around an object of interest identified by a corresponding label in an initial frame, where the object of interest appears in the frame. The object bounding box is propagated from the initial frame to subsequent frames. For the initial frame and subsequent frames, the object bounding boxes for the initial frame and subsequent frames are refined and cropped based on the associated refined object bounding boxes. A set of cropped frames is processed to determine a probability that the action to be verified is taking place from the corresponding label. In response to determining that the probability is equal to or exceeds a verification threshold, a confirmation is provided that the action-object interaction video takes place.

Description

[Technical Field]

[0001] This application relates generally to improved data processing apparatus and methods, and more particularly to mechanisms for improving action-object recognition in cluttered video scenes by utilizing text. [Background technology]

[0002] The process of training a machine learning (ML) or deep learning (DL) model involves providing an ML or DL ​​algorithm (i.e., a learning algorithm) with training data to learn from. The terms "ML model" or "DL model" refer to the model artifact produced by the training process. The training data must contain the correct answer, known as the target or target attribute. The learning algorithm finds patterns in the training data that map the input data attributes to the target (the answer you want to predict) and outputs an ML or DL ​​model that captures these patterns. Summary of the Invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described herein in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] In one exemplary embodiment, a method in a data processing system including at least one processor and at least one memory, the at least one memory including instructions executed by the at least one processor to configure the at least one processor to implement an action-object interaction detection mechanism for recognizing actions in a cluttered video scene. The method includes, in response to receiving an action-object interaction video having a corresponding label, calculating an object bounding box around an object of interest identified with the corresponding label in an initial frame of a set of frames in the action-object interaction video, where the object of interest appears in the frame. The method propagates the object bounding box from the initial frame to subsequent frames of the set of frames. For the initial frame and subsequent frames having object bounding boxes in the action-object interaction video, the method refines the object bounding boxes of the initial frame and subsequent frames using knowledge of the object of interest, thereby forming associated refined object bounding boxes, and crops the initial frame and subsequent frames based on the associated refined object bounding boxes, thereby forming a set of cropped frames. The method processes the set of cropped frames in the same order as the action-object interaction video to determine a probability that the action to be verified from the corresponding label is being performed, and then provides confirmation that the action-object interaction video performs the action to be verified in response to determining the probability that the action to be verified from the corresponding label is equal to or greater than a verification threshold.

[0005] In another exemplary embodiment, a computer program product is provided that includes a computer usable or computer readable medium having a computer readable program that, when executed on a computing device, causes the computing device to perform various and combinations of the operations outlined above with respect to the method exemplary embodiment.

[0006] In yet another exemplary embodiment, a system / apparatus is provided. The system / apparatus may include one or more processors and a memory coupled to the one or more processors. The memory may include instructions that, when executed by the one or more processors, cause the one or more processors to perform various and combinations of the operations outlined above with respect to the method exemplary embodiment.

[0007] These and other features and advantages of the present invention will be described in, or will become apparent to those skilled in the art in view of, the following detailed description of illustrative embodiments of the invention.

[0008] The invention, together with its preferred mode of use, further objects and advantages, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] 1 depicts an exemplary diagram of a distributed data processing system in which aspects of the illustrative embodiments may be implemented; [Figure 2] FIG. 1 is an exemplary block diagram of a computing device in which aspects of the illustrative embodiments may be implemented. [Figure 3] FIG. 1 is a functional block diagram of an action-object interaction detection mechanism for recognizing actions in cluttered video scenes by leveraging existing datasets, according to an exemplary embodiment. [Figure 4A]FIG. 1 illustrates an exemplary process of an action-object interaction detection mechanism that recognizes actions in cluttered video scenes by leveraging existing datasets, according to an exemplary embodiment. [Figure 4B] FIG. 1 illustrates an exemplary process of an action-object interaction detection mechanism that recognizes actions in cluttered video scenes by leveraging existing datasets, according to an exemplary embodiment. [Figure 4C] FIG. 1 illustrates an exemplary process of an action-object interaction detection mechanism that recognizes actions in cluttered video scenes by leveraging existing datasets, according to an exemplary embodiment. [Figure 4D] FIG. 1 illustrates an exemplary process of an action-object interaction detection mechanism that recognizes actions in cluttered video scenes by leveraging existing datasets, according to an exemplary embodiment. [Figure 5] 10 is an exemplary flowchart of operations performed by an action-object interaction detection mechanism in recognizing actions in a cluttered video scene by leveraging existing datasets, according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Deep learning has revolutionized computer vision. Supervised deep learning involves training complex-valued neural network models using large labeled datasets.

[0011] Large-scale labeled training datasets exist, such as ImageNet, Microsoft® Common Objects in Context (MS-COCO), and Moments in Time, for object classification, object detection, and action recognition, respectively. One drawback of deep learning models is that their performance degrades when applied to data whose feature distributions differ from those of the training data. This problem is typically addressed through transfer learning or domain adaptation-based techniques, each of which requires a significant amount of labeled training data for success. In enterprise settings, annotating client data with labels is expensive. Furthermore, client data differs from the training data in available datasets such as Moments in Time, which contains over 300 action classes, has over 1,000 examples per class, and attempts to capture real-world actions. In such scenarios, it is desirable to use neural network models pre-trained on existing datasets without the need for domain-specific adaptation.

[0012] An important class of action recognition problems from video is one in which an action is performed on an object and a text label accompanies the video. Such a text label could be, for example, an instruction such as "open a box." Knowing that the expected object is a box, it is desirable to have a way to determine whether a video shows the action "open." Correctly labeling actions occurring in a video is a key problem with applications in security, compliance verification, and more.

[0013] The illustrative embodiments provide a mechanism that utilizes a non-data-intensive approach to action recognition from videos in cluttered scenes by leveraging existing datasets and focusing on action-object interactions. Note that the terms "recognition" and "classification" are used interchangeably throughout this text. The approach leverages existing models pre-trained on large datasets to classify action-object presence in unseen data through an inference-based approach. In the presence of clutter, the mechanism focuses attention on action-object interactions.

[0014] The mechanism improves action recognition accuracy over the current state of the art in the special situation where knowledge of the objects being acted upon is available. The mechanism does not require task-specific data for fine-tuning or domain adaptation for action classification. Furthermore, the mechanism works for unseen action-object interactions resulting from state changes to objects during actions, which is important for use cases such as action compliance verification.

[0015] The strength of non-data-intensive methods comes from focusing attention on action-object interactions. Recent literature has shown that action-object interactions can be adequately learned on image datasets such as MS-COCO using methods such as RetinaNet. However, such methods and datasets are insufficient for real-world action recognition from videos. Furthermore, in real-world applications, even extensive datasets like ImageNet® are insufficient in terms of the number of examples capturing various object states, such as the state of a box when "opening" it. Furthermore, state-of-the-art action classifiers may require additional video examples for fine-tuning or domain adaptation to recognize actions being performed on objects that are not adequately covered in large-scale action classification datasets.

[0016] The mechanism of the exemplary embodiment achieves improvements over current action-object recognition by identifying objects in various states through an automated method for finding an initial seed bounding box for the object. The mechanism refines the bounding box for the object to include the context around the object rather than a dense bounding box. Object detectors, on the other hand, often output dense bounding boxes. Including the context around the object is important for improved action-object recognition over the state of the art.

[0017] Before beginning the discussion of various aspects of the exemplary embodiments, it should first be understood that throughout this description, the term “mechanism” is used to refer to elements of the present invention that perform various operations, functions, etc. As used herein, the term “mechanism” may refer to one implementation of a function or aspect of the exemplary embodiments in the form of an apparatus, a procedure, or a computer program product. In the case of a procedure, the procedure is implemented by one or more devices, apparatuses, computers, data processing systems, etc. In the case of a computer program product, the logic represented by computer code or instructions embodied in or on the computer program product is executed by one or more hardware devices to implement the function or perform the operations associated with the particular “mechanism.” Thus, the mechanisms described herein may be implemented as dedicated hardware, software running on general-purpose hardware, software instructions stored on a medium such that the instructions are readily executable by dedicated or general-purpose hardware, a procedure or method for performing a function, or any combination of the above.

[0018] The description and claims may utilize the terms "a," "at least one of," and "one or more of" in reference to particular features and elements of exemplary embodiments. It should be understood that these terms and phrases are intended to state that there is at least one of a particular feature or element present in a particular exemplary embodiment, but that more than one may be present. That is, these terms / phrases are not intended to limit the description or claims to a single feature / element present or to require the presence of more than one such feature / element. Rather, these terms / phrases only require at least a single feature / element, and more than one such feature / element may be within the scope of the description and claims.

[0019] Furthermore, it should be understood that the use of the term “engine,” when used herein in connection with describing embodiments and features of the present invention, is not intended to limit any particular implementation for achieving and / or performing the actions, steps, processes, etc., attributed to and / or performed by the engine. An engine may be, but is not limited to, software, hardware, or firmware, or any combination thereof, that performs the specified functions, including, but not limited to, any use of a dedicated processor and / or dedicated processors in combination with appropriate software loaded or stored in machine-readable memory and executed by a processor. Furthermore, any name associated with a particular engine, unless otherwise specified, is for convenience of reference and is not intended to limit it to a particular implementation. Furthermore, any functionality attributed to an engine may equally be performed by multiple engines, incorporated into and / or combined with the functionality of another engine of the same or different type, or distributed across one or more engines in various configurations.

[0020] Furthermore, it should be understood that the following description uses multiple different examples of various elements of the exemplary embodiments to further illustrate exemplary implementations of the exemplary embodiments and to aid in understanding the mechanisms of the exemplary embodiments. These examples are intended to be non-limiting and do not exhaust the various possibilities for implementing the mechanisms of the exemplary embodiments. In light of this description, it will be apparent to those skilled in the art that there are many other alternative implementations of these various elements that could be utilized in addition to, or in place of, the examples provided herein without departing from the spirit and scope of the present invention.

[0021] Accordingly, the exemplary embodiments may be utilized in many different types of data processing environments. To provide a context for the description of certain elements and functionality of the exemplary embodiments, Figures 1 and 2 are provided as example environments in which aspects of the exemplary embodiments may be implemented hereinafter. It should be understood that Figures 1 and 2 are only examples and are not intended to assert or imply any limitation with regard to the environments in which aspects or embodiments of the present invention may be implemented. Many modifications to the depicted environments may be made without departing from the spirit and scope of the present invention.

[0022] 1 illustrates a pictorial representation of an exemplary distributed data processing system in which aspects of the exemplary embodiments may be implemented. Distributed data processing system 100 may include a network of computers in which aspects of the exemplary embodiments may be implemented. Distributed data processing system 100 includes at least one network 102, which is the medium used to provide communications links between various devices and computers connected to each other within distributed data processing system 100. Network 102 may include connections such as wires, wireless communications links, fiber optic cables, etc.

[0023] In the depicted example, server 104 and server 106 are connected to network 102 along with storage unit 108. Additionally, clients 110, 112, and 114 are also connected to network 102. These clients 110, 112, and 114 may be, for example, personal computers, network computers, etc. In the depicted example, server 104 provides data such as boot files, operating system images, applications, etc. to clients 110, 112, and 114. In the depicted example, clients 110, 112, and 114 are clients to server 104. Distributed data processing system 100 may include additional servers, clients, and other devices not shown.

[0024] In the depicted example, distributed data processing system 100 is the Internet, with network 102 representing a worldwide collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, distributed data processing system 100 may also be implemented to include a number of different types of networks, such as, for example, an intranet, a local area network (LAN), a wide area network (WAN), etc. As previously mentioned, FIG. 1 is intended as an example, and not as an architectural limitation for various embodiments of the present invention, and thus the specific elements illustrated in FIG. 1 should not be considered limitations on the environments in which illustrative embodiments of the present invention may be implemented.

[0025] As shown in FIG. 1 , one or more of the computing devices, e.g., server 104, may be specifically configured to implement an action-object interaction detection mechanism for recognizing action-object actions in cluttered video scenes by leveraging existing datasets. Configuring the computing device may include providing special-purpose hardware, firmware, etc. to facilitate performing the operations and generating the outputs described herein with respect to the exemplary embodiments. Configuring the computing device may also, or alternatively, include providing software applications stored in one or more storage devices and loaded into the memory of the computing device, such as server 104, to cause one or more hardware processors of the computing device to execute software applications that configure the processors to perform the operations and generate the outputs described herein with respect to the exemplary embodiments. Furthermore, any combination of special-purpose hardware, firmware, software applications running on the hardware, etc. may be used without departing from the spirit and scope of the exemplary embodiments.

[0026] It should be understood that when a computing device is configured in one of these ways, the computing device becomes a special-purpose computing device specifically configured to implement the mechanisms of the exemplary embodiments, and not a general-purpose computing device. Moreover, as described below, implementation of the mechanisms of the exemplary embodiments improves the capabilities of computing devices and provides useful, tangible results that facilitate improved action-object recognition in cluttered video scenes, utilizing existing datasets.

[0027] As previously described, the mechanisms of the exemplary embodiments utilize a specifically configured computing device or data processing system to perform operations for recognizing action-object actions in cluttered video scenes by leveraging existing datasets. Such a computing device or data processing system may comprise various hardware elements specifically configured to implement one or more of the systems / subsystems described herein through a hardware configuration, a software configuration, or a combination of hardware and software configurations. FIG. 2 is a block diagram of just one exemplary data processing system in which aspects of the exemplary embodiments may be implemented. Data processing system 200 is an example of a computer, such as server 104 of FIG. 1, in which computer-usable code or instructions implementing processes and aspects of the exemplary embodiments of the present invention may be arranged and / or executed to achieve the operations, output, and external effects of the exemplary embodiments described herein.

[0028] In the illustrated example, data processing system 200 utilizes a hub architecture including a north bridge and memory controller hub (NB / MCH) 202 and a south bridge and input / output (I / O) controller hub (SB / ICH) 204. A processing unit 206, main memory 208, and a graphics processor 210 are connected to NB / MCH 202. Graphics processor 210 may be connected to NB / MCH 202 through an accelerated graphics port (AGP).

[0029] In the illustrated example, a local area network (LAN) adapter 212 is connected to the SB / ICH 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a hard disk drive (HDD) 226, a CD-ROM drive 230, a universal serial bus (USB) port and other communication ports 232, and PCI / PCIe devices 234 are connected to the SB / ICH 204 via bus 238 and bus 240. PCI / PCIe devices may include, for example, Ethernet adapters, add-in cards, and PC cards for notebook computers. PCI uses a card bus controller, while PCIe does not. ROM 224 may be, for example, a flash basic input / output system (BIOS).

[0030] The HDD 226 and CD-ROM drive 230 are connected to the SB / ICH 204 through a bus 240. The HDD 226 and CD-ROM drive 230 may use, for example, an integrated drive electronics (IDE) or serial advanced technology attachment (SATA) interface. A Super I / O (SIO) device 236 may be connected to the SB / ICH 204.

[0031] An operating system runs on processing unit 206. The operating system coordinates and provides control of various components within data processing system 200 of FIG. 2. As a client, the operating system may be a commercially available operating system such as Microsoft® Windows® 10®. An object-oriented programming system, such as the Java®™ programming system, may run in conjunction with the operating system to provide calls to the operating system from Java®™ programs or applications running on data processing system 200.

[0032] As a server, data processing system 200 may be, for example, an IBM eServer™ System p™ computer system running the Advanced Interactive Executive (AIX®) operating system or the Linux® operating system, a Power™ processor-based computer system, etc. Data processing system 200 may be a symmetric multiprocessor (SMP) system that includes multiple processors within processing unit 206. Alternatively, a uniprocessor system may be utilized.

[0033] Instructions for the operating system, object-oriented programming system, and applications or programs may be located on storage devices, such as HDD 226, and loaded into main memory 208 for execution by processing unit 206. Processes for exemplary embodiments of the present invention may be implemented by processing unit 206 using computer-usable program code, which may be located in a memory, such as main memory 208, ROM 224, or one or more peripheral devices 226 and 230, for example.

[0034] A bus system, such as bus 238 or bus 240 as shown in Figure 2, may comprise one or more buses. Of course, a bus system may be implemented using any type of communications fabric or architecture that provides for a transfer of data between different components or devices attached to the fabric or architecture. A communications unit, such as modem 222 or network adapter 212 of Figure 2, may include one or more devices used to transmit and receive data. A memory may be, for example, main memory 208, ROM 224, or a cache such as found in NB / MCH 202 of Figure 2.

[0035] As mentioned above, in some exemplary embodiments, the mechanisms of the exemplary embodiments may be implemented as application specific hardware, firmware, etc., application software stored in a storage device such as HDD 226 and loaded into a memory such as main memory 208 for execution by one or more hardware processors such as processing unit 206. Accordingly, the computing device shown in FIG. 2 is specifically configured to implement the mechanisms of the exemplary embodiments and is specifically configured to perform operations and generate output described below with respect to recognizing action-object actions in cluttered video scenes by leveraging existing datasets.

[0036] Those skilled in the art will appreciate that the hardware in Figures 1 and 2 may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, optical disk drives, etc., may be used in addition to or in place of the hardware depicted in Figures 1 and 2. Additionally, the processes of the illustrative embodiments may be applied to multiprocessor data processing systems other than the SMP systems described above without departing from the spirit and scope of the present invention.

[0037] Furthermore, data processing system 200 may take the form of any of a number of different data processing systems, including a client computing device, a server computing device, a tablet computer, a laptop computer, a telephone or other communications device, a personal digital assistant (PDA), etc. In some illustrative examples, data processing system 200 may be a portable computing device configured with flash memory to implement non-volatile memory for storing operating system files and / or user-generated data. In essence, data processing system 200 may be any known or later-developed data processing system without architectural limitation.

[0038] 3 shows a functional block diagram of an action-object interaction detector for recognizing actions in cluttered video scenes by leveraging existing datasets, according to an exemplary embodiment. A data processing system 300 includes an action-object interaction detector 302 that receives as input an action-object interaction video 304 including one or more frames 306a-306n and corresponding labels 308 associated with the action-object interaction video 304. The action-object interaction detector 302 outputs a determination as to whether an action occurring in the action-object interaction video 304 matches an action in the corresponding label 308. The action-object interaction detector 302 includes a label extraction engine 309, an object recognition engine 310, an object bounding box engine 312, an action classification engine 314, and a verification engine 316.

[0039] For an action-object interaction video 304 having one or more frames 306a-306n and corresponding labels 308, such as "open drawer," "close drawer," "open microwave," and "close refrigerator," the label extraction engine 309 utilizes natural language processing to first extract objects of interest, such as "drawer," "microwave," and "refrigerator," and actions to be verified, such as "open" and "close," from the corresponding labels 308. Optionally, the label extraction engine 309 then identifies a set of related objects. The need to identify related objects is motivated by the fact that the object vocabulary used in the text labels may differ from the set of object class labels associated with the image dataset on which the pre-trained object classifiers utilized by the object recognition engine 310 and object bounding box engine 312 are trained. ImageNet® is a common dataset used to pre-train object classifiers. To use an object classification model pre-trained on ImageNet® and corresponding object class labels, the object recognition engine 310 must first obtain a mapping from the object of interest extracted from the corresponding label 308 to one or more of the object class labels available in ImageNet®. For example, the commonly used ImageNet® dataset does not have an object class label corresponding to the object "drawer." For such an object, the object recognition engine 310 finds related objects using an available vocabulary database, such as WordNet®, which contains an ontology that includes synonyms and related objects for the object. In the example where the object is a "drawer," referencing the vocabulary database would provide a set of related objects, such as "filing cabinet."From this set of related objects, the object recognition engine 310 retains those present in ImageNet®. Optionally, identifying the set of related objects may be skipped if the dataset on which the pre-trained object classifier is trained contains an object of interest extracted from a corresponding label 308 among its object class labels. The object of interest extracted from the corresponding label 308 is hereinafter referred to as the object of interest or OoI. The object recognition engine 310 forms a set including the OoI, along with optional additions or substitutions to its set of related objects, whose members exist among the object class labels relevant to the dataset on which the pre-trained object classifier utilized by the object recognition engine 310 and the object bounding box engine 312 is trained. The formed set is hereinafter referred to as the classifier OoI set. According to an exemplary embodiment, unseen objects are objects for which there are insufficient views of the object in one or more rich datasets, such as ImageNet®, resulting in inaccurate predictions by the robust pre-trained object model. Such unseen object views typically result from changes in the state of an object during an action, such as the state of a "filing cabinet" during an "open" action. According to an exemplary embodiment, an unseen action-object interaction is one in which there are insufficient instances of actions being performed on the object.

[0040] As those skilled in the art will appreciate, an object classifier typically outputs a probability indicating the presence of an object for each object class for which the object classifier is trained. Similarly, well-known techniques can be used to compute a heatmap highlighting pixels that serve as identities for each object for which the object classifier outputs a probability. It is possible for a classifier OoI set to have a cardinality greater than one. Throughout this description, when the probability of an OoI appearing in a video frame or a region within a frame is determined, the probability is determined as the maximum probability exceeding the probabilities associated with one or more objects in the classifier OoI set. Similarly, when a heatmap is created that identifies pixels that serve as identities for an OoI in a video frame or a region within a frame, the object class corresponding to the maximum probability exceeding the probabilities associated with one or more objects in the classifier OoI set is used to compute the heatmap. The terms "region within a frame" and "frame region" are used interchangeably.

[0041] Using the OoI, the object recognition engine 310 processes frames from the action-object interaction video 304 to determine whether the OoI is present in one or more frames 306a-306n. Specifically, for each extracted frame, the object recognition engine 310 calculates the probability that the OoI appears in the frame using an existing object classifier trained on a dataset such as ImageNet®. The object recognition engine 310 selects the best frame as the one with the highest probability below a predetermined selection value. The object bounding box engine 312 uses the best frame identified in 310 as the best initial frame. For the best initial frame, the object bounding box engine 312 creates a heatmap that highlights pixels that serve to identify the OoI. Such a heatmap can be calculated using techniques such as the GradCam method. The object bounding box engine 312 then identifies pixels that exceed a heatmap threshold (indicating high importance for identifying the OoI). The object bounding box engine 312 fits a rectangular contour around the identified pixels, which is a bounding box for the OoI in the best initial frame.

[0042] Then, for each subsequent frame of the one or more frames 306a-306n following the identified best initial frame, the object bounding box engine 312 propagates a bounding box from the current frame to the subsequent frame. This propagation is performed using well-known techniques, such as Generic Object Tracking Using Regression Networks (GOTURN). GOTURN is a neural network-based tracker that takes as input a frame, the location of a rectangular bounding box enclosing an area of ​​interest in that frame, and a subsequent (target) frame, and predicts the location of a bounding box enclosing the same area of ​​interest in the subsequent (target) frame. Thus, as long as a pre-trained object classifier correctly classifies the OoI in at least one frame of the video, GOTURN provides a mechanism for tracking the area around the object from frame to frame. An alternative embodiment would propagate the bounding box from the current frame to subsequent frames using well-known computer vision techniques for object tracking.

[0043] The initial bounding box in each frame obtained in this way is not a dense bounding box but instead includes some context around the object. Note that the initial bounding boxes for frames subsequent to the best initial frame may be modified versions of the bounding box output by the tool used for bounding box propagation; for example, the bounding box may be scaled up or down by a given percentage. Context refinement is applied to each frame along with the bounding box determined during the bounding box determination of the best initial frame or during bounding box propagation. Context refinement is an iterative procedure. Given a video frame, in each context refinement iteration, the object bounding box engine 312 defines a frame region starting from the bounding box found so far. Such a frame region may be obtained, for example, by cropping the video frame to include only the area enclosed by the bounding box. The frame region thus obtained is input to an existing trained object classifier (trained on a dataset such as ImageNet®) to calculate the probability that an OoI appears within the frame region. The object bounding box engine 312 creates a heatmap that highlights pixels that serve as identifiers for the OoI. Such a heatmap can be calculated using techniques such as the GradCam method. The object bounding box engine 312 then identifies pixels that exceed a heatmap threshold (indicating high importance for identifying the OoI). The object bounding box engine 312 fits a rectangular contour that surrounds the identified pixels. The object bounding box engine 312 uses the rectangular contour to define a new bounding box, which is considered to be the bounding box for the current iteration.In one embodiment, the new bounding box is the rectangular contour itself. In an alternative embodiment, the new bounding box is larger than the initial bounding box for the iteration, for example, if the rectangular contour is extended to the edge of the frame domain input to the object classifier. The new bounding box then includes pixels that are in the original video frame but outside the frame domain input to the object classifier. At the end of each iteration, the new bounding box becomes the previously determined bounding box for the next iteration.

[0044] The object bounding box engine 312 iteratively refines the bounding box coordinates by jointly optimizing a cost function that measures the change in size, such as a decrease or increase, of the object bounding box in the current iteration compared to the previous bounding box, and a cost function that measures the confidence associated with the prediction that the OoI will appear within the bounding box in the current iteration. One example of a measure of the confidence of the prediction may be entropy. Jointly optimizing the two cost functions may be performed, for example, by optimizing a weighted sum of the two cost functions. The iterative refinement stops when a stopping criterion is met, for example, when the combined cost function reaches a predefined value or when the combined cost function no longer provides sufficient improvement. The context refinement procedure returns a refined bounding box, e.g., a final bounding box, or the bounding box that provides the best value of the combined cost function across the iterations. This context refinement procedure results in a bounding box that contains a useful amount of context around the object, rather than a dense bounding box. The object bounding box engine 312 then crops every frame with the refined bounding box to include only the portion of the frame within the refined bounding box, generating a cropped video frame. In one embodiment, the cropped video frames are arranged in the order in which they appear in the original video to create the cropped video.

[0045] In another embodiment, instead of propagating bounding boxes from the best initial frame to subsequent frames (one pair at a time) and then applying context refinement to each of the best initial frame and the resulting bounding boxes from the propagation, context refinement and subsequent bounding box propagation are performed sequentially for one pair of frames at a time. Specifically, context refinement is performed on the heatmap-based bounding box of the best initial frame. The resulting refined bounding box is then propagated to subsequent frames using well-known techniques, such as GOTURN. The propagated bounding box is then refined using context refinement, and the refined bounding box is propagated to the next frame, and so on.

[0046] Note that in either embodiment, a "successor" frame of the best initial frame can be a frame that follows the best initial frame in frame order within the video, a frame that precedes the best initial frame in frame order within the video (i.e., "successor" is defined in reverse video order), or both. In an exemplary embodiment, "next," "successor," and "following" are used interchangeably. Frames that do not have a refined bounding box because the context refinement procedure failed to return a bounding box (e.g., failed to identify any bounding box with an acceptably high confidence prediction that the OoI will appear within the bounding box) are removed. In other words, the frame is not included in the set of post-crop frames sent to the action classification engine 314.

[0047] The action classification engine 314 processes the post-cropped video using a pre-trained action classifier, e.g., an action classifier trained on an existing dataset such as Moments in Time. A set of action classifiers may be used rather than a single action classifier. In one embodiment, the action classifier outputs a probability that the action being performed in the video is a given action, where the given action corresponds to various action classes present in the training dataset, such as “open” or “close.” The verification engine 316 then determines whether the probability associated with the action to be verified from the corresponding label 308, such as “open” or “close,” is equal to or greater than a verification threshold. In response to the verification engine 316 determining that the probability associated with the action to be verified from the corresponding label 308 is equal to or greater than the verification threshold, the verification engine 316 provides confirmation that the action-object interaction video 304 performs the action to be verified. In an alternative embodiment, in response to the verification engine 316 determining that the probability associated with the action to be verified from the corresponding label 308 is less than the verification threshold, the verification engine 316 provides a notification that the action-object interaction video 304 failed to perform the action to be verified. In either case, the validation engine 316 may also provide a probability that the action-object interaction video 304 performs or does not perform the action to be validated. The validation threshold may be, for example, a predefined value or a predefined value adjusted based on the standard deviation of the probabilities output by an action classifier or set of action classifiers corresponding to the action classes present in the training dataset.

[0048] Note that in addition to the action to be verified, it is possible that the probability associated with one or more actions is also above the verification threshold. Ensuring that the labels present in the action classifier's training dataset do not contain synonyms is a good practice to avoid that situation. If that occurs, it is possible to maintain the above policy or, for example, modify the policy to provide confirmation that the action-object interaction video 304 performs the action to be verified from the corresponding label 308 only if the probability associated with the action to be verified from the corresponding label 308 is the highest among the probabilities associated with all of the actions in the action classifier's training dataset. This can be implemented, for example, by setting the probabilities for all but the highest class to zero.

[0049] 4A-4D illustrate an exemplary process for an action-object interaction detector, such as the action-object interaction detector 302 of FIG. 3, recognizing actions in a cluttered video scene by leveraging an existing dataset, according to an exemplary embodiment. The action-object interaction video consists of frames capturing a person closing a drawer. The input label associated with this video is "closing a drawer." The label extraction engine 309 determines the OoI as "drawer," the classifier OoI set as "filing cabinet," and the action to be verified as "close." In FIG. 4A, the action-object interaction detector identifies a best initial frame 401 from the action-object interaction video that contains an OoI 402 with the highest probability below a predetermined selection value. Once the best initial frame 401 is identified, in FIG. 4B, the action-object interaction detector calculates a heatmap of the OoI 402 in the best first frame 401, highlighting pixels that serve as the OoI's identifier. Such a heatmap may be calculated using a technique such as the GradCam approach. Pixels that exceed a heatmap threshold (indicating high importance for identifying the OoI) are identified. The object bounding box engine 312 fits a rectangular contour around the identified pixels. The rectangular contour defines a bounding box 404 around the OoI 402.

[0050] The context refinement procedure refines the bounding box 404 to generate a refined bounding box 404a for the OoI 402. That is, the action-object interaction detection mechanism iteratively refines the bounding box coordinates by jointly optimizing a cost function that measures the change in size, such as a decrease or increase, of the object bounding box in the current iteration compared to the previous bounding box, and a cost function that measures the confidence associated with the prediction that the OoI will appear within the bounding box in the current iteration. An example of a measure of the confidence of the prediction may be entropy. Jointly optimizing the two cost functions may be performed, for example, by optimizing a weighted sum of the two cost functions. The iterative refinement stops when a stopping criterion is met, for example, when the combined cost function reaches a predefined value, when the combined cost function no longer provides sufficient improvement, etc. The context refinement procedure returns a refined bounding box, e.g., a final bounding box, or the bounding box that provides the best value of the combined cost function across the iterations. This context refinement procedure results in a bounding box that contains a useful amount of context around the object, rather than a dense bounding box.

[0051] In FIG. 4C , the action-object interaction detection mechanism propagates bounding box 404 from the best initial frame 401 in FIG. 4B to frame 403 in FIG. 4C as propagated bounding box 405. A context refinement procedure refines bounding box 405 to generate refined bounding box 406 for OoI 402. That is, the action-object interaction detection mechanism iteratively refines the bounding box coordinates by jointly optimizing a cost function that measures the change in size, such as a decrease or increase, of the object bounding box in the current iteration compared to the previous bounding box, and a cost function that measures the confidence associated with the prediction that the OoI will appear within the bounding box in the current iteration. An example of a measure of the confidence of the prediction can be entropy. Jointly optimizing the two cost functions can be performed, for example, by optimizing a weighted sum of the two cost functions. The iterative refinement stops when a stopping criterion is met, e.g., when the combined cost function reaches a predefined value, or when the combined cost function no longer provides sufficient improvement. The context refinement procedure returns a refined bounding box, e.g., the final bounding box, or the bounding box that gives the best value of the combined cost function over the iterations. This context refinement procedure results in a bounding box that contains a useful amount of context around the object, rather than a dense bounding box.

[0052] In FIG. 4D , the action-object interaction detection mechanism propagates bounding box 405 from the frame in FIG. 4C to the frame in FIG. 4D to obtain bounding box 407 in FIG. 4D . A context refinement procedure refines bounding box 407 to generate bounding box 408 for OoI 402. That is, the action-object interaction detection mechanism iteratively refines the bounding box coordinates by jointly optimizing a cost function that measures the change in size, such as a decrease or increase, of the object bounding box in the current iteration compared to the previous bounding box, and a cost function that measures the confidence associated with the prediction that the OoI will appear within the bounding box in the current iteration. An example of a measure of the confidence of the prediction may be entropy. Jointly optimizing the two cost functions may be performed, for example, by optimizing a weighted sum of the two cost functions. The iterative refinement stops when a stopping criterion is met, for example, when the combined cost function reaches a predefined value or when the combined cost function no longer provides sufficient improvement. The context refinement procedure returns a refined bounding box, e.g., the final bounding box, or the bounding box that gives the best value of the combined cost function over the iterations. This context refinement procedure results in a bounding box that contains a useful amount of context around the object, rather than a dense bounding box.

[0053] The object bounding box engine 312 then crops the frame using the refined bounding boxes 404a, 406, and 408 to include only the portion of the frame within the refined bounding boxes, generating a cropped video frame. In one embodiment, the cropped video frames are arranged in the order in which they appear in the original video to create a cropped video. The action classification engine 314 processes the cropped video using a pre-trained action classifier, for example, an action classifier trained on an existing dataset such as Moments in Time. A set of action classifiers may be used rather than a single action classifier. The action classification engine then assigns probability scores to action classes, including the action class "close," and because the probability score for the action class "close" is equal to or exceeds a validation threshold, the action is verified as "close" using the validation engine 316.

[0054] The present invention may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0055] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge-in-groove structures with instructions recorded on them, and any suitable combination of the above. As used herein, computer-readable storage media should not be construed as signals that are transitory in nature, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted through wires.

[0056] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0057] Computer-readable program instructions for carrying out operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk, C++, and traditional procedural programming languages ​​such as the “C” programming language or similar. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to implement aspects of the present invention.

[0058] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0059] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, to create a machine. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular way, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0060] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0061] FIG. 5 illustrates an exemplary flowchart of operations performed by an action-object interaction detection mechanism when recognizing actions in a cluttered video scene by leveraging an existing dataset, according to an exemplary embodiment. For an identified action-object interaction video having one or more frames and corresponding labels, such as "close filing cabinet," "open microwave," or "close refrigerator," when an action begins, the label extraction engine of the action-object interaction detection mechanism extracts an initial object of interest, such as "filing cabinet," "microwave," or "refrigerator," as well as an action to be verified, such as "open" or "close," from the corresponding labels (step 502). Based on identifying the object of interest (OoI), the label extraction engine optionally identifies a set of related objects (step 504). Between the OoI and, optionally, the set of related objects, retain those present in the dataset on which the object classifier to be used in subsequent steps is trained, thereby forming a set of objects comprising the classifier object of interest (OoI) set. In step 506, all frames from the action-object interaction video are extracted. The object recognition engine then calculates the probability that the OoI appears in each frame (step 508). The object recognition engine selects the best frame as the frame with the highest probability less than a predetermined selection value and assigns it as the best initial frame (step 510).

[0062] For the best initial frame, the object recognition engine computes a heatmap highlighting pixels that serve to identify the OoI (step 512). Such a heatmap may be computed using techniques such as the GradCam method. The object recognition engine then identifies pixels that exceed a heatmap threshold (indicating high importance for identifying the OoI) (step 514). The object bounding box engine fits a rectangular contour around the identified pixels, which defines an initial bounding box for the OoI (step 516). The object bounding box engine then propagates the object bounding box from the best initial frame to the next frame, and then from that frame to the next, until bounding boxes have been determined for all frames following the best initial frame (step 518). The object bounding box engine then performs iterative context refinement on the bounding box of the best initial frame determined in step 516 and the bounding box propagated to subsequent frames in step 518 to obtain refined bounding boxes (step 520). The object bounding box engine then crops all frames for which context refinement returned refined bounding boxes and arranges them in the order in which they appear in the original video to create a cropped video (step 522).

[0063] Next, the action-object interaction detection mechanism processes the post-cropping video using a pre-trained action classifier, i.e., an action classifier trained on an existing dataset, such as Moments in Time. A set of action classifiers may be used rather than a single action classifier. The action classifier outputs the probability that the action being performed in the video is a given action, where the given action corresponds to various action classes present in the training dataset, such as “open” or “close” (step 524). The verification engine then determines whether the probability associated with the to-be-verified action, such as “open” or “close,” is greater than or equal to a verification threshold (step 526). If the verification engine determines in step 526 that the probability associated with the to-be-verified action from the corresponding label is greater than or equal to the verification threshold, the verification engine provides confirmation that the action-object interaction video performs the to-be-verified action (step 528), and the process then terminates. If the verification engine determines in step 526 that the probability associated with the action to be verified from the corresponding label is less than the verification threshold, the verification engine provides a notification that the action-object interaction video failed to perform the action to be verified (step 530), and the process then terminates. In either case, the verification engine may also provide a probability that the action-object interaction video will or will not perform the action to be verified.

[0064] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur in an order other than that described in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or a combination of dedicated hardware and computer instructions.

[0065] Thus, the illustrative embodiments provide a mechanism for action recognition in cluttered scenes by leveraging existing datasets and focusing on action-object interactions. Existing pre-trained models on large datasets are utilized to classify action-object presence in unseen data through an inference-only approach. In the presence of clutter, the mechanism focuses attention on action-object interactions. The mechanism improves action recognition accuracy over the current state of the art and does not require any task-specific data for fine-tuning or domain adaptation. Furthermore, the mechanism works on unseen action-object interactions resulting from state changes to objects during actions, which is important for use cases such as object compliance and verification.

[0066] Thus, the mechanism of the illustrative embodiments achieves an improvement over current action-object recognition by identifying the state of an object through an automated method for finding an initial seed bounding box that contains the object. The mechanism refines the bounding box that contains the object to include the context around the object, rather than the dense bounding box often found by object detectors. Including the context around the object is essential for improved action-object recognition over the state of the art.

[0067] As noted above, it should be understood that the exemplary embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment containing both hardware and software elements. In one example embodiment, mechanisms of the exemplary embodiments are implemented in software or program code, which includes, but is not limited to, firmware, resident software, microcode, etc.

[0068] A data processing system suitable for storing and / or executing program code will include at least one processor coupled directly or indirectly to memory elements, for example through a communication bus such as a system bus. The memory elements may include local memory utilized during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code must be retrieved from bulk storage during execution. The memory may be of various types, including but not limited to ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid-state memory, etc.

[0069] Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system directly or through intervening wired or wireless I / O interfaces and / or controllers, etc. I / O devices may take many different forms other than traditional keyboards, displays, pointing devices, etc., such as, for example, communication devices coupled through wired or wireless connections, including but not limited to, smart phones, tablet computers, touch screen devices, voice recognition devices, etc. Any known or later-developed I / O devices are intended to be within the scope of the exemplary embodiments.

[0070] Network adapters may also be coupled to the system to enable the data processing system to be coupled to other data processing systems, or remote printers, or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters for wired communications. Wireless communication-based network adapters may also be utilized, including, but not limited to, 802.11a / b / g / n wireless communication adapters, Bluetooth wireless adapters, and the like. Any known or later-developed network adapter is intended to be within the spirit and scope of the present invention.

[0071] The description of the present invention has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The embodiments have been chosen and described to best explain the principles, practical applications of the invention, and to enable those skilled in the art to understand the invention in various embodiments with various modifications suitable for the particular use contemplated. The terminology used herein has been chosen to best explain the principles of the embodiments, practical applications or technical improvements over technology found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method in a data processing system having a processor and a memory, the memory including instructions executed by the processor to configure the processor to implement an action-object interaction detection mechanism for recognizing actions in a cluttered video scene, the method comprising: In response to receiving an action-object interaction video having corresponding labels containing information about objects and actions related to the objects, calculating an object bounding box around an object of interest identified by the corresponding label in an initial frame of a set of frames in the action-object interaction video, wherein the object of interest appears in the frame; propagating the object bounding box from the initial frame to subsequent frames of the set of frames; For the initial frame and the subsequent frame having object bounding boxes in the action-object interaction video, using knowledge of the object of interest to refine the object bounding boxes in the initial frame and the subsequent frame, thereby forming associated refined object bounding boxes; cropping the initial frame and the subsequent frame based on the associated refined object bounding box, thereby forming a set of cropped frames; Extracting an action to be verified from the corresponding label; processing the set of cropped frames in the same order as the action-object interaction video to determine the probability that the action to be verified extracted from the corresponding label is being performed; providing a confirmation that the action-object interaction video performs the action to be verified in response to a probability that the action to be verified is being performed extracted from the corresponding label being equal to or greater than a verification threshold; and A method comprising:

2. Calculating the object bounding box around the object of interest comprises: calculating a heat map highlighting pixels that serve to identify the object of interest in the initial frame; identifying pixels that exceed a heatmap threshold; fitting a rectangular contour surrounding the identified pixels, the rectangular contour defining the object bounding box for the object of interest; The method of claim 1 , comprising:

3. Refining the object bounding box of the initial frame and the subsequent frame includes: repeatedly, cropping the frame based on the frame's associated object bounding box; calculating the probability that the object of interest will appear in the cropped frame; calculating a heat map highlighting pixels in the cropped frame that serve to identify the object of interest; identifying pixels that exceed a heatmap threshold; fitting a rectangular contour surrounding the identified pixels; determining a new object bounding box based on the rectangular outline; calculating a metric based on the size difference between the object bounding box and the new object bounding box; cropping the frame based on the new object bounding box, thereby forming a new cropped frame; calculating the probability that the object of interest appears in the new cropped frame; calculating a metric based on the probability that the object of interest will appear in the new cropped frame; calculating a cost function based on the metric capturing the size difference between the object bounding box and the new object bounding box and the metric based on the probability that the object of interest appears in the new cropped frame; evaluating a stopping criterion using the cost function; assigning the new object bounding box as the frame's associated object bounding box for the next iteration; and choosing an associated object bounding box or a new object bounding box to be the refined object bounding box for the frame when the stopping criterion is met; and The method of claim 1 , comprising:

4. The method of claim 3 , wherein the heatmap threshold indicates a high importance for identifying the object of interest.

5. the initial frame in which the object of interest appears, calculating, for each of a plurality of frames in the action-object interaction video, a probability that the object of interest appears in the frame using an object classifier; selecting a frame as the initial frame when said frame has a highest probability less than a predetermined selection value; The method of claim 1 , wherein the method comprises:

6. using the object classifier to calculate a probability that the object of interest appears in the frame; identifying a set of objects from a vocabulary database related to the object of interest identified by the corresponding label; determining a subset of the set containing the object of interest and the set of related objects, the elements of the subset corresponding to object classes of the object classifier; calculating, for each element of said subset, the probability that an object of said object class corresponding to said element will appear in said frame; determining the probability that the object of interest appears in the frame as a maximum value exceeding the probabilities associated with the elements of the subset; The method of claim 5 , comprising:

7. and in response to identifying that the probability that the action to be verified extracted from the corresponding label is being performed is less than the verification threshold, providing a notification that the action-object interaction video has failed to perform the action to be verified. The method of claim 1 further comprising:

8. providing a value of said probability together with said confirmation The method of claim 1 further comprising:

9. 2. The method of claim 1, wherein the object bounding box is propagated from the initial frame to subsequent frames of the set of frames in the action-object interaction video without first refining the bounding box through context refinement.

10. 2. The method of claim 1 , wherein after refining the bounding box through context refinement, the object bounding box is propagated from the initial frame to subsequent frames of the set of frames in the action-object interaction video.

11. The processor In response to receiving an action-object interaction video having corresponding labels containing information about objects and actions related to the objects, calculating an object bounding box around an object of interest identified by the corresponding label in an initial frame of a set of frames in the action-object interaction video, wherein the object of interest appears in the frame; propagating the object bounding box from the initial frame to subsequent frames of the set of frames; For the initial frame and the subsequent frame having object bounding boxes in the action-object interaction video, using knowledge of the object of interest to refine the object bounding boxes in the initial frame and the subsequent frame, thereby forming associated refined object bounding boxes; cropping the initial frame and the subsequent frame based on the associated refined object bounding box, thereby forming a set of cropped frames; Extracting an action to be verified from the corresponding label; processing the set of cropped frames in the same order as the action-object interaction video to determine the probability that the action to be verified extracted from the corresponding label is being performed; providing a confirmation that the action-object interaction video performs the action to be verified in response to a probability that the action to be verified is being performed extracted from the corresponding label being equal to or greater than a verification threshold; and A computer program for executing the above.

12. 1. An apparatus comprising: a processor; a memory coupled to the processor that, when executed by the processor, causes the processor to implement an action-object interaction detection mechanism for recognizing actions in a cluttered video scene, and further causes the processor to: In response to receiving an action-object interaction video having corresponding labels containing information about objects and actions related to the objects, calculating an object bounding box around an object of interest identified by the corresponding label in an initial frame of a set of frames in the action-object interaction video, wherein the object of interest appears in the frame; propagating the object bounding box from the initial frame to subsequent frames of the set of frames; For the initial frame and the subsequent frame having object bounding boxes in the action-object interaction video, using knowledge of the object of interest to refine the object bounding boxes in the initial frame and the subsequent frame, thereby forming associated refined object bounding boxes; cropping the initial frame and the subsequent frame based on the associated refined object bounding box, thereby forming a set of cropped frames; Extracting an action to be verified from the corresponding label; processing the set of cropped frames in the same order as the action-object interaction video to determine the probability that the action to be verified extracted from the corresponding label is being performed; providing a confirmation that the action-object interaction video performs the action to be verified in response to a probability that the action to be verified is being performed extracted from the corresponding label being equal to or greater than a verification threshold; and said memory containing instructions to cause An apparatus comprising:

Images