Action object recognition in cluttered video scenes using text

Abstract

A mechanism is provided to implement an action-object interaction detection mechanism for identifying actions in a cluttered video scene. An object bounding box is computed around an object of interest identified in a corresponding label in an initial frame, where the object of interest appears in the frame. The object bounding box is propagated from the initial frame to subsequent frames. For the initial frame and the subsequent frames: the object bounding boxes of the initial frame and the subsequent frames are refined and cropped based on associated refined object bounding boxes. The set of cropped frames are processed to determine a probability that an action to be verified from the corresponding label is being performed. In response to determining that the probability is equal to or exceeds a verification threshold, a confirmation is provided that the action-object interaction video performs the action to be verified.

Description

Background Technology

[0001] This application generally relates to an improved data processing apparatus and method, and more specifically, to a mechanism for improving action object recognition in cluttered video scenes by utilizing text.

[0002] The process of training a machine learning (ML) model or a deep learning (DL) model involves feeding training data to an ML or DL ​​algorithm (i.e., a learning algorithm) for learning. The terms "ML model" or "DL model" refer to the model artifact created by the training process. The training data must contain correct answers, which is called the target or target attribute. The learning algorithm looks for patterns in the training data that map the input data attributes to the target (the answer you want to predict) and outputs an ML or DL ​​model that captures these patterns. Summary of the Invention

[0003] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed description herein. This summary is not intended to identify key elements or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] In one illustrative embodiment, a method, within a data processing system including at least one processor and at least one memory, wherein the at least one memory includes instructions executed by the at least one processor to configure the at least one processor to implement an action-object interaction detection mechanism for recognizing actions in a cluttered video scene. The method includes, in response to receiving an action-object interaction video with corresponding labels, calculating object bounding boxes around objects of interest identified in the corresponding labels in an initial frame of a set of frames of the action-object interaction video, wherein the objects of interest appear in the frame. The method propagates the object bounding boxes from the initial frame to subsequent frames in the set of frames. For the initial frame and subsequent frames of the action-object interaction video with object bounding boxes: the method refines the object bounding boxes of the initial frame and subsequent frames using knowledge of the objects of interest to form associated refined object bounding boxes; and clips the initial frame and subsequent frames based on the associated refined object bounding boxes to form a set of clipped frames. The method processes the set of clipped frames in the same order as in the action-object interaction video to determine the probability that an action to be verified from the corresponding label is being performed. The method then provides confirmation that the action to be verified is performed in the action-object interaction video in response to determining the probability that the action to be verified from the corresponding tag is equal to or exceeds the verification threshold.

[0005] In other illustrative embodiments, a computer program product is provided, including a computer-usable or readable medium having a computer-readable program. When executed on a computing device, the computer-readable program causes the computing device to perform various operations and combinations thereof outlined in the illustrative embodiments of the method described above.

[0006] In another illustrative embodiment, a system / apparatus is provided. This system / apparatus may include one or more processors and memory coupled to the one or more processors. The memory may include instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations and combinations thereof outlined above in the illustrative embodiments concerning the method.

[0007] These and other features and advantages of the present invention will be described in the following detailed description of exemplary embodiments of the present invention, or will become apparent to those skilled in the art. Attached Figure Description

[0008] The invention, its preferred modes of use, and further objectives and advantages can be better understood by referring to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, wherein:

[0009] Figure 1 This is an example diagram of a distributed data processing system in which various aspects of the illustrative embodiments can be implemented;

[0010] Figure 2 This is an example block diagram of a computing device in which various aspects of the illustrative embodiments can be implemented;

[0011] Figure 3 A functional block diagram of an action-object interaction detection mechanism for identifying actions in cluttered video scenes by utilizing existing datasets, according to an illustrative embodiment, is depicted.

[0012] Figures 4A-4D An exemplary process is described in which an action-object interaction detection mechanism according to an illustrative embodiment identifies actions in a cluttered video scene by utilizing an existing dataset; and

[0013] Figure 5 An exemplary flowchart depicts the operations performed by an action-object interaction detection mechanism when identifying actions in a cluttered video scene by utilizing an existing dataset, according to an illustrative embodiment. Detailed Implementation

[0014] Deep learning has revolutionized computer vision. Supervised deep learning involves training complex neural network models using large labeled datasets.

[0015] Large labeled training datasets exist for object classification, object detection, and action recognition, such as ImageNet. Common Objects in Context (MS-COCO) and Moments in Time. One drawback of deep learning models is that their performance suffers when applied to data whose feature distribution differs from the training data. While this problem is often addressed through techniques based on transfer learning or domain adaptation, each of these techniques requires a large amount of labeled training data to succeed. In enterprise settings, labeling client-side data is expensive. Furthermore, client-side data differs from the training data in the available datasets. For example, Moments in Time contains over 300 action classes, each with over 1000 examples, and attempts to capture real-world actions. In this case, it is desirable to use a neural network model pre-trained on the existing dataset without having to perform domain-specific adaptation.

[0016] An important type of action recognition problem in videos involves performing actions on objects, with the video accompanied by text labels. These text labels can be, for example, instructions such as "open the box." The goal is to have a method to determine whether the video displays the action "open," knowing that the expected object is a box. Correctly labeling actions occurring in videos is a crucial issue in applications such as security and compliance verification.

[0017] The illustrative embodiments provide mechanisms for action recognition from videos in cluttered scenes using non-data-intensive methods, focusing on action-object interactions by leveraging existing datasets. It should be noted that the terms "recognition" and "classification" are used interchangeably throughout the text. This approach utilizes existing models pre-trained on large datasets to classify the presence of action objects in previously unseen data using an inference-based approach. In cluttered situations, these mechanisms focus on action-object interactions.

[0018] In specific environments, this mechanism improves the accuracy of action recognition beyond existing technologies, requiring knowledge of the object being operated on. These mechanisms do not require task-specific data for fine-tuning or domain adjustments to action classification. Furthermore, the mechanism operates on previously unseen action-object interactions triggered by changes in the object's state during an action, which is important for use cases such as action compliance verification.

[0019] The advantage of non-data-intensive methods lies in their focus on action-object interactions. Current literature shows that methods such as RetinaNet have been well-studied on image datasets such as MS-COCO. However, such methods and datasets are insufficient for recognizing real-world actions from videos. Furthermore, for practical applications, even methods like... Such a large dataset is also insufficient in terms of the number of examples capturing various states of objects, such as the state of a box when it is "open". Furthermore, existing action classifiers may require additional video examples for fine-tuning or domain adjustment to identify actions performed on objects that are not adequately covered in large action classification datasets.

[0020] The mechanisms of the illustrative embodiments provide improvements to current action object recognition by automating the process of finding initial seed bounding boxes for objects to identify objects in different states. These mechanisms refine the bounding boxes of objects to include the context surrounding the object, as opposed to tight bounding boxes. In contrast, object detectors typically output tight bounding boxes. Including the context surrounding the object is important for improving action object recognition compared to existing techniques.

[0021] Before discussing the various aspects of the illustrative embodiments, it should be understood that, in this specification, the term "mechanism" will be used to refer to elements that perform various operations, functions, etc., in the invention. As used herein, the term "mechanism" can refer to the implementation of a function or aspect of an illustrative embodiment in the form of an apparatus, process, or computer program product. In the case of a process, the process is implemented by one or more devices, apparatuses, computers, data processing systems, etc. In the case of a computer program product, logic represented by computer code or instructions contained in or on the computer program product is executed by one or more hardware devices to implement a function or perform an operation associated with a particular "mechanism." Therefore, the mechanisms described herein can be implemented as dedicated hardware, software executing on general-purpose hardware, software instructions stored on a medium such that the instructions are readily executable by dedicated or general-purpose hardware, a process or method for performing a function, or any combination of the foregoing.

[0022] The terms “a,” “at least one,” and “one or more” may be used in this specification and claims to refer to specific features and elements of illustrative embodiments. It should be understood that these terms and phrases are intended to indicate the presence of at least one specific feature or element in a particular illustrative embodiment, but more than one may be present. That is, these terms / phrases are not intended to limit the description or claims to the presence of a single feature / element, or to require the presence of multiple such features / elements. Rather, these terms / phrases require only at least a single feature / element, and multiple such features / elements may be within the scope of the specification and claims.

[0023] Furthermore, it should be understood that the use of the term "engine," if used herein to describe embodiments and features of the invention, is not intended to limit any particular implementation for performing and / or executing actions, steps, processes, etc., attributable to and / or performed by the engine. An engine can be, but is not limited to, software, hardware, and / or firmware, or any combination thereof, that performs a particular function, including but not limited to any combination of general-purpose and / or dedicated processors with appropriate software loaded or stored in machine-readable memory and executed by the processor. Furthermore, unless otherwise stated, any names associated with a particular engine are for convenience of reference and not for limiting any particular implementation. Moreover, any function belonging to one engine may be performed equally by multiple engines, incorporated into and / or combined with the function of another engine of the same or different type, or distributed across one or more engines in various configurations.

[0024] Furthermore, it should be understood that the following description uses multiple different instances of various elements of the illustrative embodiments to further illustrate exemplary implementations of the illustrative embodiments and to aid in understanding the mechanisms of the illustrative embodiments. These examples are intended to be non-limiting and do not exhaustively cover all possibilities for implementing the mechanisms of the illustrative embodiments. In view of this description, it will be apparent to those skilled in the art that many other alternative implementations of these various elements, in addition to or replacing the examples provided herein, can be utilized without departing from the spirit and scope of the invention.

[0025] Therefore, the illustrative embodiments can be used in many different types of data processing environments. To provide context for the description of specific elements and functions of the illustrative embodiments, Figure 1 and Figure 2 The following is provided as an example environment in which aspects of the illustrative embodiments can be implemented. It should be understood that... Figure 1 and Figure 2 This is merely an example and is not intended to assert or imply any limitation on the environment in which aspects or embodiments of the invention may be implemented. Many modifications may be made to the depicted environment without departing from the spirit and scope of the invention.

[0026] Figure 1 An illustration of an exemplary distributed data processing system is depicted, in which aspects of illustrative embodiments may be implemented. The distributed data processing system 100 may include a computer network, in which aspects of the illustrative embodiments may be implemented. The distributed data processing system 100 includes at least one network 102, which is a medium for providing communication links between various devices and computers connected together within the distributed data processing system 100. The network 102 may include connections such as wired, wireless communication links, or optical fibers.

[0027] In the depicted example, servers 104 and 106 are connected to network 102 along with storage unit 108. Clients 110, 112, and 114 are also connected to network 102. These clients 110, 112, and 114 can be, for example, personal computers, network computers, etc. In the described example, server 104 provides data, such as boot files, operating system images, and applications, to clients 110, 112, and 114. In the described example, clients 110, 112, and 114 are clients of server 104. The distributed data processing system 100 may include additional servers, clients, and other devices not shown.

[0028] In the depicted example, the distributed data processing system 100 is the Internet, where network 102 represents a global collection of networks and gateways communicating with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite. The core of the Internet is the backbone of high-speed data communication lines between master nodes or host computers, composed of thousands of commercial, government, educational, and other computer systems routing data and messages. Of course, the distributed data processing system 100 can also be implemented as including many different types of networks, such as, for example, intranets, local area networks (LANs), wide area networks (WANs), etc. As described above, Figure 1 Intended as an example, and not as an architectural limitation on different embodiments of the invention, therefore, Figure 1 The specific elements shown should not be considered as limitations on the environment in which the illustrative embodiments of the invention can be implemented.

[0029] like Figure 1 As shown, one or more computing devices, such as server 104, may be specifically configured to implement an action-object interaction detection mechanism for identifying action-object actions in cluttered video scenes by utilizing existing datasets. The configuration of the computing device may include providing dedicated hardware, firmware, etc., to facilitate the execution of operations and the generation of outputs described herein with respect to illustrative embodiments. The configuration of the computing device may also, or alternatively, include providing a software application stored in one or more storage devices and loaded into the memory of the computing device (e.g., server 104) for causing one or more hardware processors of the computing device to execute the software application, which configures the processor to perform operations and generate the outputs described herein with respect to illustrative embodiments. Furthermore, any combination of dedicated hardware, firmware, software applications executed on hardware, etc., may be used without departing from the spirit and scope of the illustrative embodiments.

[0030] It should be understood that once a computing device is configured in one of these ways, it becomes a dedicated computing device specifically configured to implement the mechanisms of the illustrative embodiments, rather than a general-purpose computing device. Furthermore, as described below, the implementation of the mechanisms of the exemplary embodiments improves the functionality of the computing device and provides useful and concrete results that help improve action object recognition in cluttered video scenes using existing datasets.

[0031] As described above, the mechanisms of the illustrative embodiments utilize specially configured computing devices or data processing systems to perform the operation of recognizing action-object actions in cluttered video scenes by leveraging existing datasets. These computing devices or data processing systems may include various hardware components that are specifically configured, through hardware configurations, software configurations, or a combination of hardware and software configurations, to implement one or more systems / subsystems described herein. Figure 2 This is a block diagram of merely one example data processing system in which various aspects of the illustrative embodiments can be implemented. Data processing system 200 is, for example... Figure 1 The example of a computer, server 104, includes computer-usable code or instructions that can be located and / or executed to implement processes and aspects of the illustrative embodiments of the invention, in order to achieve the operation, output, and external effects of the illustrative embodiments described herein.

[0032] In the depicted example, the data processing system 200 employs a hub architecture, including a Northbridge and memory controller hub (NB / MCH) 202 and a Southbridge and input / output (I / O) controller hub (SB / ICH) 204. A processing unit 206, main memory 208, and a graphics processor 210 are connected to the NB / MCH 202. The graphics processor 210 can be connected to the NB / MCH 202 via an Accelerated Graphics Port (AGP).

[0033] In the example shown, a local area network (LAN) adapter 212 is connected to SB / ICH 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a hard disk drive (HDD) 226, a CD-ROM drive 230, a universal serial bus (USB) port and other communication ports 232, and a PCI / PCIe device 234 are connected to SB / ICH 204 via buses 238 and 240. The PCI / PCIe device may include, for example, an Ethernet adapter, an add-in card, and a PC card for a laptop. PCI uses a card bus controller, while PCIe does not. ROM 224 may be, for example, a flash memory basic input / output system (BIOS).

[0034] HDD 226 and CD-ROM drive 230 are connected to SB / ICH 204 via bus 240. HDD 226 and CD-ROM drive 230 can use interfaces such as Integrated Drive Electronics (IDE) or Serial Advanced Technology Attachment (SATA). Super I / O (SIO) device 236 can be connected to SB / ICH 204.

[0035] The operating system runs on the processing unit 206. The operating system coordinates and controls... Figure 2 The data processing system 200 contains various components. As a client, the operating system can be a commercially available operating system, such as... Windows Object-oriented programming systems, such as Java TM The programming system can run alongside the operating system and provides Java execution on the data processing system 200. TM Calls made by a program or application to the operating system.

[0036] As a server, the data processing system 200 can, for example, run advanced interactive execution. Operating system or IBM eServer operating system TM System Computer systems, Power-based TM Computer systems with processors, etc. The data processing system 200 can be a symmetric multiprocessor (SMP) system that includes multiple processors in the processing unit 206. Alternatively, a single-processor system can be used.

[0037] The operating system, object-oriented programming system, and instructions for applications or programs reside on a storage device (such as HDD 226), can be loaded into main memory 208, and executed by processing unit 206. The processes of the illustrative embodiments of the invention can be executed by processing unit 206 using computer-usable program code, which may reside in memory such as main memory 208, ROM 224, or in one or more peripheral devices 226 and 230.

[0038] Bus systems, such as Figure 2 The bus 238 or bus 240 shown may include one or more buses. Of course, the bus system can be implemented using any type of communication structure or architecture that provides data transmission between different components or devices attached to that structure or architecture. Such as Figure 2The communication unit of the modem 222 or network adapter 212 may include one or more devices for sending and receiving data. The memory may be, for example, main memory 208, ROM 224, or something similar. Figure 2 The cache found in NB / MCH 202.

[0039] As described above, in some illustrative embodiments, the mechanism of the illustrative embodiments can be implemented as dedicated hardware, firmware, etc., application software stored in a storage device (such as HDD 226) and loaded into a memory (such as main memory 208), executed by one or more hardware processors (such as processing unit 206), etc. Thus, Figure 2 The computing device shown is specifically configured to implement the mechanisms of the illustrative embodiments and is specifically configured to perform operations by utilizing existing datasets and generate outputs as described below regarding the identification of action-object actions in cluttered video scenes.

[0040] Those skilled in the art should understand that Figure 1 and Figure 2 The hardware can vary depending on the implementation. Besides... Figure 1 and 2 In addition to the hardware shown, or instead Figure 1 and 2 The hardware shown can be replaced with other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives. Furthermore, without departing from the spirit and scope of the invention, the processes of the exemplary embodiments can be applied to multiprocessor data processing systems, rather than the aforementioned SMP systems.

[0041] Furthermore, the data processing system 200 can take the form of a variety of different data processing systems, including client computing devices, server computing devices, tablet computers, laptops, telephones or other communication devices, personal digital assistants (PDAs), etc. In some illustrative examples, the data processing system 200 may be a portable computing device configured with flash memory to provide non-volatile storage for storing, for example, operating system files and / or user-generated data. Essentially, the data processing system 200 can be any known or later-developed data processing system without architectural limitations.

[0042] Figure 3A functional block diagram of an action-object interaction detection mechanism, according to an illustrative embodiment, for identifying actions in cluttered video scenes by utilizing existing datasets is depicted. The data processing system 300 includes an action-object interaction detection mechanism 302 that receives an action-object interaction video 304 and corresponding labels 308 associated with the action-object interaction video 304 as input. The action-object interaction video 304 includes one or more frames 306a-306n. The action-object interaction detection mechanism 302 outputs a determination regarding whether an action occurring in the action-object interaction video 304 matches an action in the corresponding label 308. The action-object interaction detection mechanism 302 includes a label extraction engine 309, an object recognition engine 310, an object bounding box engine 312, an action classification engine 314, and a verification engine 316.

[0043] For an action-object interaction video 304 with one or more frames 306a-306n and corresponding labels 308, such as "open drawer", "close drawer", "open microwave", "close refrigerator", etc., the label extraction engine 309 uses natural language processing to initially extract objects of interest, such as "drawer", "microwave", "refrigerator", etc., from the corresponding labels 308, as well as actions to be verified, such as "open" and "close". Optionally, the label extraction engine 309 then identifies a set of related objects. The need to identify related objects is driven by the fact that the vocabulary of objects used in text labels may differ from a set of object class labels associated with an image dataset, on which a pre-trained object classifier used by the object recognition engine 310 and the object bounding box engine 312 is trained. This is a general dataset used for pre-training object classifiers. In order to use... Given a pre-trained object classification model based on the corresponding object class labels, the object recognition engine 310 must first obtain the object of interest extracted from the corresponding label 308. A mapping of one or more object class tags available in the library. For example, commonly used... The dataset does not have an object class label corresponding to the object "drawer". For such objects, the object recognition engine 310 uses, for example, a label such as... The available vocabulary database is used to find relevant objects. This vocabulary database contains an ontology, which includes synonyms of the object and related objects. In the example where the object is a "drawer," the reference vocabulary database will provide a set of related objects, such as "file cabinet." From this set of related objects, the object recognition engine 310 retains the existing... The objects in the dataset on which the pre-trained object classifier is trained contain objects of interest extracted from their corresponding labels 308 in their object class labels. The identification of a set of related objects can be optionally skipped. Objects of interest extracted from their corresponding labels 308 will be referred to hereinafter as objects of interest or OoIs. The object recognition engine 310 forms a set containing OoIs, wherein its associated object set is optionally added to or replaced, wherein members of this set exist in the object class labels associated with the dataset on which the pre-trained object classifier used by the object recognition engine 310 and the object bounding box engine 312 is trained. The resulting set will be referred to hereinafter as the classifier OoI set. According to the illustrative embodiment, unseen objects are objects that are present in one or more rich datasets (such as...). The lack of a sufficient view of the object leads to incorrect predictions from a robust pre-trained object model. This unseen object view is typically caused by state changes of the object during an action, such as the state of the "file cabinet" during the "open" action. According to the illustrative embodiment, an unseen action-object interaction is an interaction in which there are not enough examples of actions being performed on the object.

[0044] As those skilled in the art will understand, object classifiers typically output probabilities indicating the presence of objects of each object class for which they are trained. Similarly, using known techniques, a heatmap can be computed that highlights the pixels responsible for identifying each object for which the object classifier outputs probabilities. The cardinality of the classifier OoI set may be greater than 1. Throughout this specification, when the probability of an OoI appearing in a video frame or a region within a frame is determined, that probability is determined as the highest probability among the probabilities associated with one or more objects in the classifier OoI set. Similarly, when creating a heatmap that identifies the pixels responsible for identifying OoIs of a video frame or a region within that frame, the object class corresponding to the highest probability among the probabilities associated with one or more objects in the classifier OoI set is used to compute the heatmap. The terms “region in frame” and “frame region” are used interchangeably.

[0045] Using OoIs, the object recognition engine 310 processes frames from the action-object interaction video 304 and determines whether an OoI exists in one or more frames 306a-306n. Specifically, for each extracted frame, the object recognition engine 310 calculates the probability that the OoI appears in that frame, for example, using... An existing object classifier is trained on the dataset. Object recognition engine 310 selects the best frame as the frame with the highest probability less than a predetermined selection value. Object bounding box engine 312 uses the best frame identified in 310 as the best initial frame. For the best initial frame, object bounding box engine 312 creates a heatmap that highlights pixels responsible for identifying OoIs. This heatmap can be computed using techniques such as the GradCam method. Object bounding box engine 312 then identifies pixels that exceed a heatmap threshold (indicating high importance for OoI identification). Object bounding box engine 312 fits a rectangular outline surrounding the identified pixels. The rectangular outline is the bounding box of the OoI in the best initial frame.

[0046] Then, for each subsequent frame 306a-306n following the identified best initial frame, the object bounding box engine 312 propagates the bounding boxes from the current frame to the subsequent frames. This propagation is accomplished using known techniques, such as, for example, General Object Tracking (GOTURN) using a regression network. GOTURN is a neural network-based tracker that takes the frame, the position of the rectangular bounding box surrounding the region of interest within that frame and subsequent (target) frames as input, and predicts the position of the bounding box surrounding the same region of interest in subsequent (target) frames. Thus, GOTURN provides a mechanism to track the region around an object frame by frame, provided that a pre-trained object classifier correctly classifies the OoI in at least one frame of the video. An alternative is to use known computer vision techniques for object tracking to propagate the bounding boxes from the current frame to subsequent frames.

[0047] The initial bounding boxes obtained in this way in each frame are not tight bounding boxes, but rather contain some context surrounding the object. Note that the initial bounding boxes of frames after the optimal initial frame can be modified versions of the bounding boxes output by tools used for bounding box propagation; for example, the bounding boxes can be expanded or shrunk by a given percentage. During the determination of the bounding boxes in the optimal initial frame or during bounding box propagation, context refinement is applied to each frame with the determined bounding boxes. Context refinement is an iterative process. Given a video frame, in each context refinement iteration, the object bounding box engine 312 starts from the bounding boxes determined so far to define a frame region. This frame region can be obtained, for example, by cropping the video frame to include only the region enclosed by the bounding boxes. The frame regions obtained in this way are then fed into an existing trained object classifier (such as...). The object bounding box engine 312 is trained on a dataset to calculate the probability that Col appears in the frame region. The engine creates a heatmap that highlights pixels responsible for OoI identification. This heatmap can be computed using techniques such as, for example, the GradCam method. The object bounding box engine 312 then identifies pixels that exceed a heatmap threshold (indicating high importance for OoI identification). The object bounding box engine 312 fits a rectangular outline surrounding the identified pixels. The object bounding box engine 312 uses the rectangular outline to define a new bounding box, which is considered the bounding box for the current iteration. In one embodiment, the new bounding box is the rectangular outline itself. In an alternative embodiment, the new bounding box may be larger than the initial bounding box of the iteration, for example, if the rectangular outline extends to the edge of the frame region input to the object classifier. The new bounding box then includes pixels located within the original video frame but outside the frame region input to the object classifier. At the end of each iteration, the new bounding box becomes the bounding box determined so far for the next iteration.

[0048] The object bounding box engine 312 iteratively refines the bounding box coordinates by jointly optimizing a cost function (which measures the change in the size of the object bounding box in the current iteration compared to previous bounding boxes, such as a decrease, increase, etc.) and a cost function (which measures the confidence associated with the prediction that the OoI appears in the bounding box of the current iteration). An example of prediction confidence is entropy. For example, the two cost functions can be jointly optimized by optimizing a weighted sum of the two cost functions. Iterative refinement stops when stopping criteria are met, such as when the combined cost function reaches a predetermined value, when the combined cost function stops improving sufficiently, etc. The context refinement process returns refined bounding boxes, such as the final bounding box or the bounding box that gives the best value of the combined cost function in the iteration. The bounding boxes produced by this context refinement process include a useful amount of context around the object, rather than a tight bounding box. The object bounding box engine 312 then crops each frame with the refined bounding boxes to include only the frame portion within the refined bounding boxes, which produces cropped video frames. In an embodiment, the cropped video frames are placed in the order in which they appear in the original video to create the cropped video.

[0049] In another embodiment, instead of propagating the bounding boxes from the optimal initial frame to subsequent frames (pair by pair) and then applying context refinement to the optimal initial frame and each bounding box generated by the propagation, context refinement and subsequent bounding box propagation are performed one pair of frames at a time, one after another. Specifically, context refinement is done on the heatmap-based bounding boxes of the optimal initial frame. Then, the refined bounding boxes are propagated to subsequent frames, for example using a known technique such as GOTURN. The propagated bounding boxes are then refined using context refinement, and the refined bounding boxes are propagated to the next frame, and so on.

[0050] Please note that in any embodiment, a frame “following” the best initial frame can be a frame after the best initial frame in the frame order of the video, a frame before the best initial frame in the frame order of the video (i.e., “following” is defined in reverse video order), or both. The illustrative embodiments may use “next,” “following,” and “subsequent” interchangeably. Frames without refined bounding boxes are discarded because the context refinement process failed to return bounding boxes (e.g., because it failed to identify any bounding boxes with an acceptablely high confidence prediction, i.e., OoIs appearing within the bounding boxes). In other words, they are not included in the set of cropped frames sent to the action classification engine 314.

[0051] Action classification engine 314 processes the cropped video using a pre-trained action classifier, such as an action classifier trained on an existing dataset, like Moments in Time. A set of action classifiers can be used instead of a single action classifier. In an embodiment, one or more action classifiers output the probability that the action being performed in the video is a given action, where the given action corresponds to various action classes present in the training dataset, such as "on," "off," etc. Verification engine 316 then determines whether the probability associated with the action to be verified from the corresponding label 308, such as "on," "off," etc., is equal to or exceeds a verification threshold. In response to verification engine 316 determining that the probability associated with the action to be verified from the corresponding label 308 is equal to or exceeds the verification threshold, verification engine 316 provides confirmation that action-object interaction video 304 has performed the action to be verified. Alternatively, in response to verification engine 316 determining that the probability associated with the action to be verified from the corresponding label 308 is below the verification threshold, verification engine 316 provides notification that action-object interaction video 304 failed to perform the action to be verified. In either case, the verification engine 316 can also provide the probability that the action-object interaction video 304 performs or does not perform the action to be verified. The verification threshold can be, for example, a predefined value or a predefined value adjusted based on the standard deviation of the probabilities output by an action classifier or a set of action classifiers corresponding to the action classes present in the training dataset.

[0052] Note that, in addition to the action to be verified, the probability associated with one or more actions may also be equal to or exceed the verification threshold. Ensuring that the labels present in the action classifier's training dataset do not contain synonyms is good practice to avoid this situation. If this does occur, it is possible to maintain or modify the above strategy, for example, to provide confirmation of the action to be verified from the corresponding label 308 only if the probability associated with the action to be verified from the corresponding label 308 is the highest among all probabilities associated with actions in the action classifier's training dataset. For example, this could be achieved by setting the probability of all classes except the highest-ranking class to zero.

[0053] Figures 4A-4D An exemplary process is described according to an illustrative embodiment of an action-object interaction detection mechanism that identifies actions in cluttered video scenes by utilizing existing datasets, such as... Figure 3 Action-object interaction detection mechanism 302. The action-object interaction video includes frames capturing the action of a person closing a drawer. The input label associated with this video is "closing drawer". Label extraction engine 309 determines the OoI as "drawer", sets the classifier OoI to "filing cabinet", and determines the action to be verified as "closing". Figure 4A In the process, the action-object interaction detection mechanism identifies an optimal initial frame 401 from the action-object interaction video. This optimal initial frame 401 includes the OoI 402 with the highest probability of being less than a predetermined selection value. With the optimal initial frame 401 identified, in... Figure 4B In this process, the action object interaction detection mechanism calculates a heatmap of the best OoI 402 within the first frame 401, highlighting the pixels responsible for identifying the OoI. This heatmap can be calculated using techniques such as, for example, the GradCam method. Pixels exceeding a heatmap threshold are identified (indicating high importance for OoI identification). The object bounding box engine 312 fits a rectangular outline surrounding the identified pixels. The rectangular outline defines the bounding box 404 around the OoI 402.

[0054] The context refinement process refines the bounding box 404 to produce a refined bounding box 404a for the OoI 402. That is, the action-object interaction detection mechanism iteratively refines the bounding box coordinates by jointly optimizing a cost function (which measures the change in the size of the object's bounding box in the current iteration compared to previous bounding boxes, e.g., decreasing, increasing, etc.) and a cost function (which measures the confidence associated with the prediction that the OoI appears in the bounding box of the current iteration). An example of prediction confidence is entropy. For example, the two cost functions can be jointly optimized by optimizing a weighted sum of the two cost functions. Iterative refinement stops when stopping criteria are met, e.g., when the combined cost function reaches a predetermined value, when the combined cost function stops improving sufficiently, etc. The context refinement process returns the refined bounding box, such as the final bounding box or the bounding box that gives the optimal value of the combined cost function in the iteration. The bounding boxes produced by this context refinement process include a useful amount of context around the object, rather than a tightly packed bounding box.

[0055] exist Figure 4C In the middle, the action-object interaction detection mechanism removes the bounding box 404 from... Figure 4B The best initial frame 401 in the propagation to Figure 4CFrame 403 in the context is used as the propagation bounding box 405. The context refinement process refines the bounding box 405 to produce a refined bounding box 406 for OoI 402. That is, the action-object interaction detection mechanism iteratively refines the bounding box coordinates by jointly optimizing a cost function (which measures the change in the size of the object bounding box in the current iteration compared to previous bounding boxes, such as decreasing, increasing, etc.) and a cost function (which measures the confidence associated with the prediction that the OoI appears in the bounding box of the current iteration). An example of prediction confidence is entropy. For example, the two cost functions can be jointly optimized by optimizing a weighted sum of the two cost functions. Iterative refinement stops when stopping criteria are met, such as when the combined cost function reaches a predetermined value, when the combined cost function stops improving sufficiently, etc. The context refinement process returns the refined bounding box, such as the final bounding box or the bounding box that gives the best value of the combined cost function in the iteration. The bounding boxes produced by this context refinement process include a useful amount of context around the object, rather than a tight bounding box.

[0056] exist Figure 4D In the middle, the action-object interaction detection mechanism will Figure 4C The bounding box 405 propagates to Figure 4D In the border, to obtain Figure 4D Bounding box 407. Context refinement process – Refines bounding box 407 to produce bounding box 408 for OoI 402. That is, the action-object interaction detection mechanism iteratively refines the bounding box coordinates by jointly optimizing a cost function (which measures the change in the size of the object bounding box in the current iteration compared to previous bounding boxes, e.g., decrease, increase, etc.) and a cost function (which measures the confidence associated with the prediction that the OoI appears in the bounding box of the current iteration). An example of prediction confidence is entropy. For example, the two cost functions can be jointly optimized by optimizing a weighted sum of the two cost functions. Iterative refinement stops when stopping criteria are met, e.g., when the combined cost function reaches a predetermined value, when the combined cost function stops improving sufficiently, etc. The context refinement process returns the refined bounding box, such as the final bounding box or the bounding box that gives the best value of the combined cost function in the iteration. The bounding boxes produced by this context refinement process include a useful amount of context around the object, rather than a tight bounding box.

[0057] Then, the object bounding box engine 312 crops the frames with thinned bounding boxes 404a, 406, and 408 to include only the portions of the frames within the thinned bounding boxes, resulting in cropped video frames. In this embodiment, the cropped video frames are placed in the order they appear in the original video to create the cropped video. The action classification engine 314 uses a pre-trained action classifier to process the cropped video, such as an action classifier trained on an existing dataset, like Moments in Time. A set of action classifiers can be used instead of a single action classifier. The action classification engine then assigns probability scores to action classes, including the action class "Off," and because the probability score for the action class "Off" is equal to or exceeds a validation threshold, the validation engine 316 validates the action as "Off."

[0058] This invention can be a system, method, and / or computer program product. A computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0059] Computer-readable storage media can be tangible devices that retain and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punched cards or raised structures in recesses on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being a transient signal, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0060] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or to an external computer or external storage device, via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0061] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute the computer-readable program instructions by utilizing the status information of the computer-readable program instructions to personalize the electronic circuitry, thereby performing aspects of the invention.

[0062] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0063] These computer-readable program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute through the processor of the computer or other programmable data processing apparatus, can create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks. These computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other equipment to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of manufacture comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0064] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operational steps to be performed on the computer, other programmable apparatus or other equipment, thereby producing a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus or other equipment implement the functions / actions specified in the flowchart and / or block diagram.

[0065] Figure 5 An exemplary flowchart depicts the operations performed by an action-object interaction detection mechanism when identifying actions in a cluttered video scene using an existing dataset, according to an illustrative embodiment. When the operation begins, for an identified action-object interaction video with one or more frames and corresponding labels (such as “close filing cabinet,” “open microwave,” “close refrigerator,” etc.), the label extraction engine of the action-object interaction detection mechanism extracts initial objects of interest (OoIs) from the corresponding labels, such as “filing cabinet,” “microwave,” “refrigerator,” etc., and the actions to be verified, such as “open,” “close,” etc. (step 502). Based on the identified OoIs, the label extraction engine optionally identifies a set of related objects (step 504), and among the OoIs and the optional set of related objects, retains those objects present in the dataset, on which an object classifier to be used in subsequent steps is trained, thereby forming a set of objects including the set of classifier objects of interest (OoIs). In step 506, all frames from the action-object interaction video are extracted. The object recognition engine then calculates the probability of an OoI appearing in each frame (step 508). The object recognition engine selects the best frame as the frame with the highest probability of being less than the predetermined selection value and designates it as the best initial frame (step 510).

[0066] For the optimal initial frame, the object recognition engine computes a heatmap (step 512) that highlights the pixels responsible for identifying the OoI. This heatmap can be computed using techniques such as, for example, the GradCam method. The object recognition engine then identifies pixels exceeding a heatmap threshold (indicating high importance for OoI identification) (step 514). The object bounding box engine fits a rectangular outline surrounding the identified pixels, defining the initial bounding box of the OoI (step 516). The object bounding box engine then propagates the object bounding boxes from the optimal initial frame to the next frame, and then from that frame to the next, until bounding boxes have been determined for all frames after the optimal initial frame (step 518). The object bounding box engine then performs iterative context refinement on the bounding boxes of the optimal initial frame determined in step 516 and the bounding boxes propagated to subsequent frames in step 518 to obtain refined bounding boxes (step 520).

[0067] The object bounding box engine then refines the crop context, returning all frames of the refined bounding boxes and placing them in the order they appeared in the original video to create the cropped video (step 522).

[0068] Next, the action-object interaction detection mechanism uses a pre-trained action classifier, i.e., an action classifier trained on an existing dataset, such as Moments in Time, to process the cropped video. A set of action classifiers can be used instead of a single action classifier. One or more action classifiers output the probability that the action being performed in the video is a given action, where the given action corresponds to various action classes present in the training dataset, such as "on", "off", etc. (step 524). The verification engine then determines whether the probability associated with the action to be verified, such as "on", "off", etc., is equal to or exceeds a verification threshold (step 526). If, in step 526, the verification engine determines that the probability associated with the action to be verified from the corresponding label is equal to or exceeds the verification threshold, the verification engine provides confirmation that the action-object interaction video has performed the action to be verified (step 528), after which the process ends. If, in step 526, the verification engine determines that the probability associated with the action to be verified from the corresponding label is below the verification threshold, the verification engine provides notification that the action-object interaction video has failed to perform the action to be verified (step 530), after which the process ends. In either case, the verification engine may also provide the probability that the action-object interaction video has performed or not performed the action to be verified.

[0069] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, fragment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may not occur in the order indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0070] Therefore, illustrative embodiments provide mechanisms for action recognition in cluttered scenarios focusing on action-object interactions by leveraging existing datasets. Using existing pre-trained models on large datasets, the presence of action objects is classified into previously unseen data through an inference-only approach. In cluttered situations, these mechanisms focus attention on action-object interactions. These mechanisms improve the accuracy of action recognition at the state-of-the-art level and do not require any task-specific data for fine-tuning or domain adjustment. Furthermore, these mechanisms work on previously unseen action-object interactions that are caused by changes in the object's state during the action process, which is important for use cases such as object conformity and verification.

[0071] Therefore, the mechanism of the illustrative embodiment provides an improvement in current action object recognition by identifying the state of the object via an automated method for finding an initial seed bounding box that includes the object. This mechanism refines the bounding box that includes the object to include the context surrounding the object, as opposed to the tight bounding boxes typically detected by object detectors. Including the context surrounding the object is essential for improved action object recognition compared to existing technologies.

[0072] As described above, it should be understood that illustrative embodiments may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment that includes both hardware and software elements. In one example embodiment, the mechanisms of the illustrative embodiment are implemented in software or program code, including but not limited to firmware, resident software, microcode, etc.

[0073] A data processing system suitable for storing and / or executing program code includes at least one processor that is directly or indirectly connected to a storage element via a communication bus (such as a system bus). The memory element may include local memory, mass storage, and cache memory used during the actual execution of the program code. The cache memory provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from the mass storage during execution. The memory can be of various types, including but not limited to ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid-state memory, etc.

[0074] Input / output (I / O) devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system directly or via intermediate wired or wireless I / O interfaces and / or controllers. Besides traditional keyboards, displays, pointing devices, etc., I / O devices can take many different forms, such as communication devices coupled via wired or wireless connections, including but not limited to smartphones, tablets, touchscreen devices, voice recognition devices, etc. Any known or hereafter developed I / O devices are within the scope of these exemplary embodiments.

[0075] Network adapters can also be connected to the system to enable the data processing system to connect to other data processing systems or remote printers or storage devices via an intermediate private or public network. Modems, cable modems, and Ethernet cards are just a few types of network adapters currently available for wired communication. Wireless communication-based network adapters can also be used, including but not limited to 802.11a / b / g / n wireless communication adapters, Bluetooth wireless adapters, etc. Any known or subsequently developed network adapters are within the spirit and scope of this invention.

[0076] The description of this invention is for illustrative and descriptive purposes only and is not intended to be exhaustive or limiting of the forms disclosed herein. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. These embodiments were chosen and described in order to best explain the principles of the invention, its practical application, and to enable others skilled in the art to understand the various embodiments of the invention with various modifications to suit a particular intended use. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or technical improvements to techniques found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method in a data processing system including at least one processor and at least one memory, wherein the at least one memory includes instructions executable by the at least one processor to configure the at least one processor to implement an action-object interaction detection mechanism for recognizing actions in a cluttered video scene, the method comprising: In response to receiving an action-object interaction video with a corresponding label, calculate the object bounding box around the object of interest identified in the corresponding label in the initial frame of a set of frames of the action-object interaction video, wherein the object of interest appears in the frame; Propagate the object bounding box from the initial frame to subsequent frames in the set of frames; For the initial frame and subsequent frames with object bounding boxes in the action-object interaction video: The object bounding boxes of the initial frame and the subsequent frames are refined using knowledge of the object of interest, thereby forming associated refined object bounding boxes; as well as The initial frame and the subsequent frames are cropped based on the associated refined object bounding box to form a set of cropped frames; The set of cropped frames is processed in the same order as in the action-object interaction video to determine the probability that the action to be verified from the corresponding tag is being performed; as well as In response to determining the probability that the action to be verified from the corresponding tag is equal to or exceeds a verification threshold, confirmation is provided that the action-object interaction video performs the action to be verified.

2. The method according to claim 1, wherein, Calculating the bounding box of the object around the object of interest includes: Calculate a heatmap, which highlights pixels responsible for identifying the object of interest in the initial frame; Identify pixels that exceed the heatmap threshold; and Fit a rectangular outline that surrounds the identified pixels, wherein the rectangular outline defines the object bounding box of the object of interest.

3. The method according to claim 1, wherein, Refining the object bounding boxes of the initial frame and the subsequent frames includes: Iteratively: The frame is cropped based on the associated object bounding box of the frame; Calculate the probability that the object of interest appears in the cropped frame; Calculate a heatmap that highlights pixels in the cropped frame responsible for identifying the object of interest; Identify pixels that exceed the heatmap threshold; Fit a rectangular outline that surrounds the identified pixels; A new object bounding box is determined based on the rectangular outline; The metric is calculated based on the size difference between the object bounding box and the new object bounding box; The frame is cropped based on the new object bounding box to form a new cropped frame; Calculate the probability that the object of interest appears in the new cropped frame; The metric is calculated based on the probability that the object of interest appears in the new cropped frame; The cost function is calculated based on a metric of the size difference between the captured object bounding box and the new object bounding box, and a metric based on the probability that the object of interest appears in the new cropped frame. Use the aforementioned cost function to evaluate the stopping criteria; and Assign the new object bounding box as the associated object bounding box of the frame for the next iteration; and When the stopping criteria are met, the associated object bounding box or a new object bounding box is selected as the refined object bounding box of the frame.

4. The method according to claim 3, wherein, The heatmap threshold indicates the high importance of the object of interest.

5. The method according to claim 1, wherein, The initial frame in which the object of interest appears in the frame is identified by a method including the following: For each of the multiple frames in the action-object interaction video, an object classifier is used to calculate the probability that the object of interest appears in the frame; as well as The frame is selected as the initial frame when it has the highest probability of being less than a predetermined selection value.

6. The method according to claim 5, wherein, Calculating the probability that the object of interest appears in the frame using the object classifier includes: Identify a set of objects from the vocabulary database that are related to the objects of interest identified in the corresponding tags; Determine a subset comprising the object of interest and a set of related objects, wherein the elements of the subset correspond to the object classes of the object classifier; For each element of the subset, calculate the probability that an object of the object class corresponding to that element appears in the frame; and The probability of the object of interest appearing in the frame is determined as the maximum of the probabilities associated with the elements of the subset.

7. The method according to claim 1, further comprising: In response to the probability that the action to be verified from the corresponding tag is below a verification threshold, a notification is provided that the action-object interaction video failed to perform the action to be verified.

8. The method according to claim 1, further comprising: The probability value is provided for the confirmation.

9. The method according to claim 1, wherein, The object bounding box is propagated from the initial frame in the action-object interaction video to subsequent frames in the set of frames, without first refining the bounding box through contextual refinement.

10. The method according to claim 1, wherein, After refining the bounding box through contextual refinement, the object bounding box is propagated from the initial frame to subsequent frames of the set of frames in the action-object interaction video.

11. A computer program product comprising a computer-readable storage medium storing a computer-readable program therein, wherein the computer-readable program, when executed on a computing device, causes the computing device to implement an action-object interaction detection mechanism for recognizing actions in a cluttered video scene, and further causes the computing device to: In response to receiving an action-object interaction video with a corresponding label, calculate the object bounding box around the object of interest identified in the corresponding label in the initial frame of a set of frames of the action-object interaction video, wherein the object of interest appears in the frame; Propagate the object bounding box from the initial frame to subsequent frames in the set of frames; For the initial frame and subsequent frames with object bounding boxes in the action-object interaction video: The object bounding boxes of the initial frame and the subsequent frames are refined using knowledge of the object of interest, thereby forming associated refined object bounding boxes; as well as The initial frame and the subsequent frames are cropped based on the associated refined object bounding box to form a set of cropped frames; The set of cropped frames is processed in the same order as in the action-object interaction video to determine the probability that the action to be verified from the corresponding tag is being performed; as well as In response to determining the probability that the action to be verified from the corresponding tag is equal to or exceeds a verification threshold, confirmation is provided that the action-object interaction video performs the action to be verified.

12. The computer program product according to claim 11, wherein, A computer-readable program for calculating the bounding box of objects around the object of interest also enables the computing device to: Calculate a heatmap, which highlights pixels responsible for identifying the object of interest in the initial frame; Identify pixels that exceed the heatmap threshold; and Fit a rectangular outline that surrounds the identified pixels, wherein the rectangular outline defines the object bounding box of the object of interest.

13. The computer program product according to claim 11, wherein, The computer-readable program for refining the object bounding boxes of the initial frame and the subsequent frames also enables the computing device to: Iteratively: The frame is cropped based on the associated object bounding box of the frame; Calculate the probability that the object of interest appears in the cropped frame; Calculate a heatmap that highlights pixels in the cropped frame responsible for identifying the object of interest; Identify pixels that exceed the heatmap threshold; Fit a rectangular outline that surrounds the identified pixels; A new object bounding box is determined based on the rectangular outline; The metric is calculated based on the size difference between the object bounding box and the new object bounding box; The frame is cropped based on the new object bounding box to form a new cropped frame; Calculate the probability that the object of interest appears in the new cropped frame; The metric is calculated based on the probability that the object of interest appears in the new cropped frame; The cost function is calculated based on a metric of the size difference between the captured object bounding box and the new object bounding box, and a metric based on the probability that the object of interest appears in the new cropped frame. Use the aforementioned cost function to evaluate the stopping criteria; and Assign the new object bounding box as the associated object bounding box of the frame for the next iteration; as well as When the stopping criteria are met, the associated object bounding box or a new object bounding box is selected as the refined object bounding box of the frame.

14. The computer program product according to claim 13, wherein, The heatmap threshold indicates the high importance of the object of interest.

15. The computer program product according to claim 11, wherein, The initial frame in which the object of interest appears in the frame is identified by the computer-readable program, which further causes the computing device to: For each of the multiple frames in the action-object interaction video, an object classifier is used to calculate the probability that the object of interest appears in the frame; as well as The frame is selected as the initial frame when it has the highest probability of being less than a predetermined selection value.

16. The computer program product according to claim 15, wherein, A computer-readable program for calculating the probability that the object of interest appears in the frame using the object classifier also enables the computing device to: Identify a set of objects from the vocabulary database that are related to the objects of interest identified in the corresponding tags; Determine a subset comprising the object of interest and a set of related objects, wherein the elements of the subset correspond to the object classes of the object classifier; For each element of the subset, calculate the probability that an object of the object class corresponding to that element appears in the frame; as well as The probability of the object of interest appearing in the frame is determined as the maximum of the probabilities associated with the elements of the subset.

17. The computer program product according to claim 11, further comprising: In response to the probability that the action to be verified from the corresponding tag is below a verification threshold, a notification is provided that the action-object interaction video failed to perform the action to be verified.

18. The computer program product according to claim 11, wherein, The computer-readable program also enables the computing device to: The probability value is provided for the confirmation.

19. The computer program product according to claim 11, wherein: The object bounding box is propagated from the initial frame in the action-object interaction video to subsequent frames in the set of frames, without first refining the bounding box through contextual refinement. or After refining the bounding box through contextual refinement, the object bounding box is propagated from the initial frame to subsequent frames of the set of frames in the action-object interaction video.

20. An apparatus comprising: processor; as well as A memory coupled to the processor, wherein the memory includes instructions that, when executed by the processor, cause the processor to implement an action-object interaction detection mechanism for recognizing actions in cluttered video scenes, and further cause the processor to: In response to receiving an action-object interaction video with a corresponding label, calculate the object bounding box around the object of interest identified in the corresponding label in the initial frame of a set of frames of the action-object interaction video, wherein the object of interest appears in the frame; Propagate the object bounding box from the initial frame to subsequent frames in the set of frames; For the initial frame and subsequent frames with object bounding boxes in the action-object interaction video: The object bounding boxes of the initial frame and the subsequent frames are refined using knowledge of the object of interest, thereby forming associated refined object bounding boxes; as well as The initial frame and the subsequent frames are cropped based on the associated refined object bounding box to form a set of cropped frames; The set of cropped frames is processed in the same order as in the action-object interaction video to determine the probability that the action to be verified from the corresponding tag is being performed; as well as In response to determining the probability that the action to be verified from the corresponding tag is equal to or exceeds a verification threshold, confirmation is provided that the action-object interaction video performs the action to be verified.

Images