Automated interaction activity recognition based on keypoint detection

Abstract

The computer-based automated interaction activity recognition based on keypoint detection includes extracting a time sequence of image frames from a video recording by one or more processors. The one or more processors use machine learning techniques to identify first and second keypoints in each of the image frames in the time sequence. The first keypoints are associated with objects in the time sequence of image frames, and the second keypoints are associated with individuals interacting with the objects. The one or more processors combine the first keypoints with the second keypoints, extract spatiotemporal features from the combination, and use the spatiotemporal features to train a classification model capable of recognizing interaction activity.

Description

[Technical field]

[0001] The present invention relates generally to the field of artificial intelligence (AI), and more particularly to a method, system, and computer program product for automated interactive activity recognition based on keypoint detection using machine learning techniques. [Background technology]

[0002] Interaction activity recognition systems are important for computers to understand the actions of people in photos or videos. These systems can be used in many fields. For example, in manufacturing and maintenance processes, interaction activity recognition systems can be implemented to improve the use of certain machines by identifying operation errors. However, the interaction activities between humans and objects are complex, and recognizing the relevant actions can be a difficult task. Summary of the Invention

[0003] The shortcomings of the prior art are overcome and additional advantages are provided by providing a method for interaction activity recognition. The method includes extracting, by one or more processors, a time sequence of image frames from a video recording, and identifying, by the one or more processors, first and second keypoints in each of the image frames in the time sequence. The first keypoints are associated with an object in the time sequence of the image frames, and the second keypoints are associated with an individual interacting with the object. The one or more processors combine the first keypoints with the second keypoints and extract spatio-temporal features from the combined first and second keypoints. The one or more processors train a classification model for recognizing interaction activity between the individual and the object based on the extracted spatio-temporal features.

[0004] Another embodiment of the present disclosure provides a computer program product for interaction activity recognition based on the above-mentioned method.

[0005] Another embodiment of the present disclosure provides a computer system for interaction activity recognition based on the above-mentioned method.

[0006] The following detailed description, given by way of example and not by way of limitation, is best understood in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating a networked computer environment according to one embodiment of the present disclosure. [Figure 2A] FIG. 1 illustrates a system for automated interaction activity recognition based on keypoint detection according to one embodiment of the present disclosure. [Figure 2B] FIG. 2B illustrates an exemplary implementation of a system for automated interaction activity recognition based on keypoint detection of FIG. 2A according to one embodiment of the present disclosure. [Diagram 3] 1 is a flowchart illustrating steps of a computer-implemented method for automated interaction activity recognition based on keypoint detection according to one embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram of internal and external components of a computer system according to one embodiment of the present disclosure. [Diagram 5] FIG. 1 is a block diagram of an exemplary cloud computing environment, according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a block diagram of functional layers of the exemplary cloud computing environment of FIG. 5 in accordance with one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The drawings are not necessarily to scale. The drawings are merely schematic and are not intended to portray specific parameters of the invention. The drawings are intended to depict only typical embodiments of the invention. In the drawings, like numbering represents like elements.

[0009] Although detailed embodiments of the claimed structures and methods are disclosed herein, it will be understood that the disclosed embodiments are merely exemplary of the claimed structures and methods, which may be embodied in various forms. However, the present invention may be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. The description may omit details of well-known features and techniques so as not to unnecessarily obscure the presented embodiments.

[0010] The recognition of complex interaction activities performed by humans plays a key role in many applications, e.g., human-robot interaction monitoring systems, security, sports biomechanics, etc. An interaction activity recognition system must be able to automatically analyze everyday activities and detect human-machine interactions that may be undesirable in order to provide an appropriate response. Such undesirable interactions may include interactions that may be dangerous, e.g., when a human is operating a machine.

[0011] Many of these applications can benefit from spatial localization of features of interest. These features of interest are often referred to as "keypoints." Keypoint detection typically involves the simultaneous detection of spatial locations or points in an image that define the features of interest, or salient features in an image at a given time (i.e., spatiotemporal features). Desirable properties of keypoint detection can include accurate localization; invariance to shift, rotation, scale, and illumination changes; robustness to noise; high recall; etc.

[0012] However, since humans routinely interact with different types of objects in different ways depending on their intentions (e.g., the tasks to be performed), the detection and recognition of the performed interaction activities becomes a very challenging task.

[0013] Accordingly, embodiments of the present invention provide methods, systems, and computer program products for automatically recognizing interaction activity based on keypoint detection. Exemplary embodiments described below provide, among other things, systems, methods, and computer program products for capturing a sequence of image frames from a video recording, detecting keypoints associated with interaction activity in each image frame using machine learning techniques, combining the detected keypoints (e.g., object keypoints and human keypoints), using the combined keypoints as input for a graph convolutional network that is used to determine spatiotemporal features, and using the determined spatiotemporal features to train a classification model capable of recognizing and classifying interaction activity.

[0014] Thus, embodiments of the present invention have the ability to improve the technical field of artificial intelligence by providing a system and computer-implemented method that uses deep learning algorithms to automatically recognize and classify interaction activities from available video recordings. Embodiments of the present invention may accurately recognize interactions between humans and specific objects to determine actions that can be improved. For example, equipment in a manufacturing facility that is misoperated may cause operational or safety issues. Other examples may include the analysis of body movements related to sports to optimize a player's skills or techniques. In both situations, the proposed method for automated interaction activity recognition may collect and analyze data, with prior user approval, to detect suboptimal human mechanics and generate notifications that allow corrective actions to be taken.

[0015] Referring to Figure 1, an exemplary networked computing environment 100 is shown in accordance with one embodiment of the present disclosure. Figure 1 is provided only as an illustration of one embodiment and is not intended to suggest any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made by one skilled in the art without departing from the scope of the invention as set forth in the claims.

[0016] The networked computing environment 100 may include a client computer 102 and a communications network 110. The client computer 102 may include a data storage device 106 and a processor 104 capable of executing an interaction activity recognition program 108. The client computer 102 may be, for example, a mobile device, a phone (including a smartphone), a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of accessing a network.

[0017] The networked computing environment 100 may also include a server computer 114 having a data storage device 120 and a processor 118 capable of executing the software program 112. In some embodiments, the server computer 114 may be a resource management server, a web server, or any other electronic device capable of transmitting and receiving data. In other embodiments, the server computer 114 may represent a server computing system that utilizes multiple computers as a server system, such as in a cloud computing environment.

[0018] The interaction activity recognition program 108 executing on the client computer 102 may communicate with a software program 112 executing on a server computer 114 via a communications network 110. As described with reference to FIG. 4, the client computer 102 and the server computer 114 may include internal and external components.

[0019] The networked computing environment 100 may include multiple client computers 102 and server computers 114, only one of which is shown. The communications network 110 may include various types of communications networks according to embodiments of the present disclosure, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a public switched telephone network (PSTN), a cellular or mobile data network (e.g., wireless Internet provided by third or fourth generation cellular mobile communications), a private branch exchange (PBX), any combination thereof, or any combination of connections and protocols supporting communications between the client computers 102 and the server computers 114. The communications network 110 may include wired, wireless, or fiber optic connections. As known to those skilled in the art, the networked computing environment 100 may include additional computing devices, servers, or other devices not shown.

[0020] For components, operations, or structures described herein as a single instance, multiple instances may be provided. Boundaries between various components, operations, and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific exemplary configurations. Other allocations of functionality are contemplated and may fall within the scope of the invention. In general, structures and functions presented as separate components in the exemplary configurations may be implemented as combined structures or components. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements may be included within the scope of the invention.

[0021] Referring to Fig. 2A, a system 200 for automated interaction activity recognition based on keypoint detection according to one embodiment of the present disclosure is shown. Fig. 2B is an exemplary implementation of the system 200 described in conjunction with Fig. 2A.

[0022] According to one embodiment, the system 200 for automated interaction activity recognition includes an image capture module 212, a keypoint detection module 214, a feature extraction module 218, and an interaction activity classification module 220. The image capture module 212 captures videos or images from an optical instrument. In one embodiment, the optical instrument can be a video camera or the like. Specifically, the image capture module 212 is capable of retrieving a sequence of image frames 240 (FIG. 2B) from a video recording captured by the optical instrument (not shown). The process of retrieving the sequence of image frames 240 from the recorded video can be performed using any known image processing tool. For example, a software library such as OpenCv® (an open source computer vision library) that converts images into multidimensional arrays can be used to retrieve the sequence of image frames 240.

[0023] It should be noted that any data collection (e.g., photos, videos, etc.) is done with the user's consent, using opt-in and opt-out features. As known to those skilled in the art, opt-in and opt-out features generally relate to ways in which a user can change their participation status (i.e., consent or decline data collection). In some embodiments, the opt-in and opt-out features may include software applications available on the client computer 102. Additionally, a user may choose to stop the collection or use of their information. In some embodiments, the user may be notified each time data is collected. It is envisioned that the collected data is protected and will not be shared with anyone without the user's consent. The user may stop data collection at any time.

[0024] After extracting the sequence of image frames 240 from the video, the keypoint detection module 214 identifies features or points of interest (i.e., keypoints) associated with people and objects present in the sequence of image frames 240. Specifically, the keypoint detection module identifies keypoints associated with objects 250 in each image frame from the retrieved sequence of image frames 240. Subsequently, the keypoint detection module 214 identifies keypoints associated with people (or individuals) 260 in each image frame from the retrieved sequence of image frames 240.

[0025] According to one embodiment, a convolutional neural network (CNN) pre-trained to determine object interest points can be used to determine object keypoints 252. As known to those skilled in the art, a CNN is a class of deep learning neural networks for image recognition and classification that can analyze an input (i.e., an image) and output a class or a probability that the input is a particular class. In one embodiment, a CNN-based real-time object detection model, such as YOLO(C) (You Only Look Once) and / or Faster R-CNN, can be used to detect a bounding box of a particular object (e.g., object 250) in a sequence of image frames 240 and identify object keypoints 252. The CNN-based detection model can be trained to identify object keypoints using pre-labeled data.

[0026] Similarly, existing techniques can be used to determine human keypoints 262. For example, real-time methods for multi-person pose detection in images and videos, such as OpenPose, can be used to determine human keypoints 262. As known to those skilled in the art, these real-time multi-person detection methods can detect human body, hand, face, and foot keypoints together on a single image.

[0027] In some cases, occluded keypoints may be present in the sequence of image frames 240 (e.g., keypoints that are difficult to see or not captured by the camera) and go unnoticed by conventional keypoint determination systems. To avoid this problem, the system 200 uses the before and after of each image frame in the sequence of image frames 240 to determine existing occluded (human or object) keypoints. Specifically, the keypoint detection module uses a tracking algorithm in the keypoint detection process to determine the relationship between the nth frame and the (n+1)th frame in the sequence of image frames 240. If a human or object keypoint is occluded in the (n+1)th frame, the location of the occluded keypoint can be determined based on the location of the same keypoint in the nth frame where the keypoint is visible or unoccluded.

[0028] The object keypoints 252 and human keypoints 262 are combined and used as input for the feature extraction module 218. The feature extraction module 218 consists of a Graph Convolutional Neural Network (GCN) that extracts spatio-temporal features that can be used to train a classification model for the interaction activity classification module 220. The spatio-temporal features relate to the collected data across both space and time that correspond to the determined object and human keypoints 252, 262, respectively.

[0029] As known to those skilled in the art, GCN provides an image classification method that includes a very powerful neural network architecture for machine learning on graphs. Specifically, GCN performs convolutions on graphs rather than images composed of pixels. For example, a CNN model extracts the most important information from a sequence of image frames 240 to classify the sequence, while a GCN model passes filters on the graph to look for important vertices and edges (i.e., keypoints) that can help classify nodes in the graph. Specifically, GCN models can handle non-Euclidean structure data (e.g., images) more efficiently and provide more application space on topologically connected data structures. The results from the GCN model include the final object and human keypoints, which are used as inputs for the classification model of the interaction activity classification module 220.

[0030] The extracted spatio-temporal features are input into a classification model of the interaction activity classification module 220. According to one embodiment, the interaction activity classification module 220 outputs or generates an optimal classification of interactions between humans and objects based on the extracted spatio-temporal features. The classification model of the interaction activity classification module 220 uses a supervised learning algorithm that learns from the input data (i.e., spatio-temporal features associated with keypoints extracted from the sequence of image frames 240) and uses this learning to classify new interaction activities between humans and objects.

[0031] In some embodiments, the classification model of the interaction activity classification module 220 can recognize undesired interaction activity between humans and objects based on a training dataset and generate alerts to inform about unusual interaction actions.

[0032] Referring to FIG. 3, a flowchart illustrating steps of a computer-implemented method for automated interaction activity recognition based on keypoint detection is shown, according to one embodiment of the present disclosure.

[0033] The process begins in step 302 by receiving a number of image frames. As explained above, the number of image frames corresponds to a sequence of image frames extracted from a video recording. In step 304, a CNN-based model is used to detect first keypoints corresponding to interest points associated with an object (e.g., object 250 and object keypoints 252 in FIG. 2B). According to one embodiment, the CNN-based model is pre-trained to recognize the first (object) keypoints.

[0034] Similarly, second keypoints corresponding to interest points associated with a human or individual interacting with the object are detected within the received multiple image frames in step 306. The second or human keypoints are detected using state-of-the-art real-time methods for multi-person pose detection in images and videos.

[0035] In embodiments where occluded or hidden keypoints are present, the proposed method uses the before and after of each image frame to determine the existing occluded (human and / or object) keypoints, as described above with reference to FIG. 2A.

[0036] In step 308, the first (object) keypoint and the second (human) keypoint are combined and used as the input of the GCN model of the feature extraction module 218 of FIG. 2A in step 310. Specifically, in step 308, a predetermined time window N is set, and the first and second keypoints corresponding to the time window N are selected, combined and used as the input of the GCN model in step 310. The output of the GCN model for the corresponding time window N may then be used to train a classification model (e.g., the classification model of the interaction activity classification module 220 of FIG. 2A). In step 312, recognition of the interaction activity between the human and the object can be performed based on the trained classification model. In other words, the output of the GCN model (i.e., the associated spatiotemporal features of the combined keypoints) is used to feed into the (interaction) classification model, as described above with reference to FIG. 2A.

[0037] In some embodiments, the proposed method can generate alerts informing of detected unusual interactions between humans and objects based on recognizing the interaction activity as unwanted or suboptimal, which can be particularly useful in manufacturing processes or security systems.

[0038] Thus, embodiments of the present invention provide a method, system, and computer program product for recognizing and classifying interaction activities between humans and objects based on keypoints detected from a time sequence of image frames. The proposed embodiments use machine learning algorithms to accurately recognize and classify human and object keypoints, which can then be used to train a classification model for interaction activity recognition. The proposed embodiments may be useful in many practical applications where recognition of activities between humans and objects is required, including manufacturing processes, security systems, maintenance systems, etc.

[0039] Referring to Figure 4, a block diagram of client computer 102 and server computer 114 components of the networked computing environment 100 of Figure 1 is shown in accordance with one embodiment of the present disclosure. It should be understood that Figure 4 is only intended to provide an illustration of one implementation and is not intended to suggest any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

[0040] The client computer 102 and the server computer 114 may include one or more processors 402, one or more computer-readable RAMs 404, one or more computer-readable ROMs 406, one or more computer-readable storage media 408, device drivers 412, read / write drives or interfaces 414, and network adapters or interfaces 416, all interconnected through a communications fabric 418. The communications fabric 418 may be implemented with any architecture designed to pass data and / or control information between processors (e.g., microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components in the system.

[0041] One or more operating systems 410 and one or more application programs 411 are stored in one or more of the computer readable storage media 408 for execution by one or more of the processors 402 through one or more of the respective RAMs 404 (which typically include cache memory). In the illustrated embodiment, each of the computer readable storage media 408 may be an internal hard drive magnetic disk storage device, a CD-ROM, a DVD, a memory stick, a magnetic tape, a magnetic disk, an optical disk, a semiconductor storage device, such as a RAM, a ROM, an EPROM, a flash memory, or any other computer readable tangible storage device capable of storing computer programs and digital information.

[0042] The client computer 102 and the server computer 114 may also include a R / W drive or interface 414 for reading from and writing to one or more portable computer readable storage media 426. Application programs 411 on the client computer 102 and the server computer 114 may be stored on one or more of the portable computer readable storage media 426 and read via the respective R / W drive or interface 414 and loaded into the respective computer readable storage media 408.

[0043] The client computer 102 and the server computer 114 may also include a network adapter or interface 416, such as a TCP / IP adapter card or a wireless communication adapter (such as a 40 wireless communication adapter using OFDMA technology), for connection to a network 428. Application programs 411 on the client computer 102 and the server computer 114 may be downloaded to the computing device from an external computer or external storage device via a network (e.g., the Internet, a local area network, or other wide area or wireless network) and the network adapter or interface 416. Programs may be loaded from the network adapter or interface 416 into the computer-readable storage medium 408. The network may include copper wire, optical fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or combinations thereof.

[0044] The client computer 102 and the server computer 114 may also include a display screen 420, a keyboard or keypad 422, and a computer mouse or touchpad 424. The device driver 412 interfaces to the display screen 420 for imaging, to the keyboard or keypad 422, to the computer mouse or touchpad 424, or to the display screen 420 for alphanumeric entry and pressure sensing of user selections, or a combination thereof. The device driver 412, the R / W drive or interface 414, and the network adapter or interface 416 may include hardware and software (stored in the computer readable storage medium 408 or ROM 406, or both).

[0045] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0046] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0047] Its features are as follows.

[0048] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as they need it, without the need for human interaction with the provider of the service.

[0049] Broad Network Access: Capabilities are available over the network and are accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).

[0050] Resource Pooling: Pooling provider computing resources to serve multiple consumers using a multi-tenant model where various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control over or knowledge of the exact location of the resources provided to them, although there is a sense of location independence in that the location may be identifiable at a higher level of abstraction (e.g. country, state, or data center).

[0051] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out and rapidly release to quickly scale in. To the consumer, provisionable capacity often appears to be unlimited and can be purchased in any quantity at any time.

[0052] Metered Services: Cloud systems leverage metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of the services utilized.

[0053] The service model is as follows:

[0054] Software as a Service (SaaS): The capability offered to the consumer is to use the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer has no management or control over the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application features, with the possible exception of limited user-specific application configuration settings.

[0055] Platform as a Service (PaaS): The capability offered to a consumer is to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and possibly the application hosting environment configuration.

[0056] Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0057] The deployment model is as follows:

[0058] Private Cloud: The cloud infrastructure is operated exclusively for an organization. It can be managed by the organization or a third party and can be on-premise or off-premise.

[0059] Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organization or a third party and may exist on-premise or off-premise.

[0060] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0061] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0062] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0063] Referring now to FIG. 5, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped (not shown), for example, in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure-as-a-service, platform-as-a-service, and / or software-as-a-service services without the need for cloud consumers to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N illustrated in FIG. 5 are intended to be exemplary only, and that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0064] Referring now to Figure 6, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 5) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 6 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0065] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (reduced instruction set computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0066] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.

[0067] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and accounting or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources such that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance arrangement and procurement of cloud computing resources anticipated to be required in the future according to SLAs.

[0068] The Workload Layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and a system for automated interaction activity recognition based on keypoint detection 96.

[0069] The programs described herein are identified based on the applications in which they are implemented in specific embodiments of the invention. However, it should be understood that any specific program names herein are used for convenience only, and thus the invention should not be limited to use in any particular application identified and / or implied by such names.

[0070] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions described in the blocks may occur out of the order shown. For example, depending on the functionality involved, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may possibly be executed in the reverse order. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0071] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0072] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or groove ridge structures having instructions recorded thereon, and any suitable combination thereof. A computer-readable storage medium, as used herein, should not be construed to be a transitory signal per se, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over a wire.

[0073] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device over a network, such as, for example, the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0074] The computer readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the last scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing state information of the computer readable program instructions to execute the computer readable program instructions to perform aspects of the present invention.

[0075] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0076] These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that, when the instructions are executed by the processor of the computer or other programmable data processing apparatus, means are generated for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored constitutes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0077] Furthermore, computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps, thereby generating a computer-implemented process that, when executed on the computer, other programmable apparatus, or other device, implements the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0078] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order shown. For example, depending on the functionality involved, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may possibly be executed in the reverse order. It will also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0079] Although steps of the disclosed methods and components of the disclosed systems and environments have been identified sequentially or consecutively using numbers and letters, such numbering or lettering does not indicate that such steps must be performed in the order recited, but is merely provided to facilitate clear reference to the method steps. Additionally, steps of the present methods may be performed in parallel to perform their described functions.

[0080] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for interaction activity recognition, comprising: by one or more processors, Extracting a time sequence of image frames from the video recording; identifying a first keypoint in each of the image frames in the time sequence, the first keypoint associated with an object in the time sequence of image frames; identifying second keypoints in each of the image frames in the time sequence, the second keypoints associated with an individual interacting with the object; Combining the first keypoint with the second keypoint; extracting spatio-temporal features from the combined first and second keypoints, the extracting including using the combined first and second keypoints as inputs to a Graph Convolutional Neural Network (GCN) model for extracting the spatio-temporal features; training a classification model for recognizing an interaction activity between the individual and the object based on the extracted spatio-temporal features; 4. A computer-implemented method comprising:

2. Identifying the first keypoint comprises: using a convolutional neural network (CNN) based detection model trained by the one or more processors to identify the first keypoint. The method of claim 1 further comprising:

3. using, by the one or more processors, the pre-labeled data to train the CNN-based detection model to identify the first keypoint. The method of claim 2 , further comprising:

4. Identifying the second keypoints includes: Using, by said one or more processors, a real-time method for multi-person pose detection in images and videos. The method of claim 1 further comprising:

5. The method of claim 1, wherein the results of the GCN model include a final first keypoint and a final second keypoint.

6. feeding, by the one or more processors, the spatio-temporal features to the classification model. The method of claim 1 further comprising:

7. identifying, by the one or more processors, a third keypoint that corresponds to the occluded keypoint. The method of claim 1 further comprising:

8. Identifying the third keypoint comprises: determining, by the one or more processors, a relationship between an nth frame and an (n+1)th frame from the time sequence of image frames using a tracking algorithm, wherein a position of the third keypoint in the (n+1)th frame is determined based on a position of the third keypoint in the nth frame in which the third keypoint is visible; The method of claim 7 further comprising:

9. 1. A computer system for interaction activity recognition, comprising: One or more processors; the one or more processors: Extracting a time sequence of image frames from the video recording; identifying a first keypoint in each of the image frames in the time sequence, the first keypoint associated with an object in the time sequence of image frames; identifying second keypoints in each of the image frames in the time sequence, the second keypoints associated with an individual interacting with the object; Combining the first keypoint with the second keypoint; extracting spatio-temporal features from the combined first and second keypoints, the extracting including using the combined first and second keypoints as inputs to a Graph Convolutional Neural Network (GCN) model for extracting the spatio-temporal features; training a classification model for recognizing an interaction activity between the individual and the object based on the extracted spatio-temporal features; A computer system that executes the above.

10. A program for causing a processor to carry out a computer implemented method according to any one of claims 1 to 8.

Images