Method and system for micro-activity identification
A system using synchronized cameras and AI modules addresses the challenge of monitoring microactivities by accurately identifying and scoring them, improving task quality and safety through real-time feedback.
Patent Information
- Application Number
- JP2024078603
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-05-14
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2044-05-14
AI Technical Summary
Existing methods struggle to accurately monitor and score microactivities in complex tasks due to resource-intensive video processing and the lack of focus on task-related microactivities, especially in environments where human intervention is necessary, leading to potential quality and safety issues.
A system utilizing synchronized video and audio cameras, combined with a pre-trained AI model comprising modules for object-hand association, video classification, audio analysis, hand gesture recognition, pose estimation, optical character recognition, and dimension measurement, to identify and score microactivities based on spatial and temporal information, ensuring correct tool usage and handling.
The system effectively monitors and scores microactivities in real-time, providing detailed feedback on task quality and safety, enhancing operational efficiency and reducing errors by detecting and penalizing deviations from established procedures.
Smart Images

Figure 2024164829000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS REFERENCE AND PRIORITY TO RELATED APPLICATIONS This application claims priority to Indian Patent Application No. 202321034111 filed on May 15, 2023.
[0002] TECHNICAL FIELD The disclosure herein relates generally to microactivity identification, and more particularly to a computer-implemented method and system for monitoring a task for identification and quality scoring of task-related microactivities. [Background technology]
[0003] In real-world environments, some activities / tasks are highly complex, such as industrial processes, automated assembly lines, medical procedures involving surgery, etc. Sometimes, these complex processes involve the intervention of skilled experts for their execution. Achieving quality within a process depends on the performance of the person performing it. Although automation is considered a promising option to maintain quality within a process, in some cases, it is not possible to have a fully automated system. This may be due to infrastructure issues, or sometimes, in a production or assembly line or some procedures, there are stages where human intervention is mandatory to ensure certain quality standards. Even when a task is performed by a robot, the robot actions can be analyzed based on close monitoring of their actions. Furthermore, improvements can be brought about in robotic assembly through learning from human actions or from other robots. Despite the implementation of lean manufacturing concepts and standard operating procedures (SOPs), sometimes, workers / operators tend to intentionally or unintentionally overlook the SOPs, which results in inferior work execution and ultimately worsens the quality of the work. This is a normal tendency for humans to make mistakes in complex industrial environments. To ensure quality, an operator / worker should perform a set of actions as per SOP for example in a production line and the product must be assembled without any defects. Following an established process for handling a task not only ensures quality but also provides checks and balances to ensure safety in the work environment. The safety of the process as well as the safety of the worker performing the task ensures reduction and minimization of hazards that may arise due to oversight of the established SOP.
[0004] Solutions implemented in the state of the art (SOTA) have proposed a rational way to visualize information about human actions for assembly and quality in the production line, allowing the management of defect information in a way that is directed to the worker / operator performing certain activities. Other solutions have implemented acoustic quality control systems with satisfactory results in their production lines, as well as machine learning systems to identify deviations from SOPs. However, the monitoring or supervision of events occurring in real environments where there is no control over the worker's working style and the sequence of assembly processes remains unaddressed. Such events occurring in the real world involve many microactivities that should be performed through human intervention. A microactivity may be defined as a set of activities that occur while performing a task. A microactivity lasts for a short period of time (just a few seconds to a few seconds), for example, while performing a task of assembling a car engine, and involves various microactivities such as picking up the right screw, holding the screw correctly, properly fastening the screw, etc. Such microactivities involve gestures, postures, and movements of body parts. Video surveillance is an effective mechanism to supervise or inspect activities or microactivities. During supervision, activity through a video capture mechanism requires capturing all of those micro-activities of high importance by monitoring the viewpoint. However, processing the captured video to automatically detect task- or event-related micro-activities remains resource intensive. A significant amount of computational power is required to handle the huge amount of output generated by complex computer processing. In addition, each deployment site usually has only limited space and resources to handle all micro-activity related video streams. Existing approaches offer few solutions that focus only on task-related micro-activities. Summary of the Invention [Means for solving the problem]
[0005] The embodiments of the present disclosure present technical improvements as solutions to one or more of the above technical problems recognized by the inventors in conventional systems. For example, in one embodiment, a method of micro-activity identification is provided. The method includes receiving a video stream via a plurality of synchronized video and audio camera devices. The plurality of synchronized video and audio camera devices cover a task under surveillance from various angles. The method further includes decoding the video stream by a capture device, where the capture device decodes the video stream into a plurality of frames to extract associated audio from the video stream. The video stream of the person performing the task under surveillance is provided to the capture device for extracting a batch of frames from the video stream. The frame extraction results in the extraction of both spatial and temporal information. The batch of frames is used for video estimation. The number of frames in the batch is configurable and may be adjusted according to the type of micro-activity monitored. The method further includes detecting at least one micro-activity in the plurality of frames by a pre-trained AI model. The pre-trained AI model includes an object-hand-association (OHA) module, a video classification module, an audio analysis (AA) module, a hand gesture (HG) module, a pose estimation (PE) module, an optical character recognition (OCR) module, a dimension measurement (DM) module, and an occupied hand (OH) module. The ODA module is based on deep learning (DL) methods involving neural networks, and the most important feature of DL is that it automatically extracts the features to be focused on for analysis during the learning process. The output of each module is mapped to the exact duration of the micro-activity performed.The method further includes a step of validating the micro-activities detected by the pre-trained AI model, where the validation involves matching the detected micro-activities with a ground truth sequence previously fed to the AI model. A condition check is performed during the validation. For example, if the predicted result of the video classification module matches the ground truth list, no penalty is imposed and further activities are checked for correct tool usage and correct tool handling. However, if the predicted result of the video classification module does not match the ground truth list, a penalty is imposed and a certain pre-defined score is subtracted. The method further includes a step of assigning a weighting to each of the micro-activities detected by the multiple modules, a step of scoring the task by adding a positive score to each correctly performed micro-activity, and a step of assigning a penalty to the incorrectly performed micro-activity. The weighting assigned to each micro-activity is based on the correct tool usage and correct tool handling while performing the micro-activity. The method further includes a step of obtaining a quality score for the task based on the individual scores assigned to the multiple detected micro-activities.
[0006] In another aspect, a system for micro-activity identification is provided. The system includes at least one memory storing programmed instructions, one or more input / output (I / O) interfaces, and one or more hardware processors, and the AI model includes a number of modules operatively coupled to the corresponding at least one memory, namely, an object-hand association (OHA) module, a video classification module, an audio analysis (AA) module, a hand gesture (HG) module, a pose estimation (PE) module, an optical character recognition (OCR) module, a dimension measurement (DM) module, and an occupied hand (OH) module, and the system is configured to receive a video stream via a number of synchronized video and audio camera devices. The number of synchronized video and audio camera devices cover the task under supervision from different angles. Further, the system is configured to decode the video stream by a capture device, which decodes the video stream into a number of frames to extract associated audio from the video stream. The video stream of the person performing the task under supervision is provided to the capture device for extracting a batch of frames from the video stream. The frame extraction results in the extraction of both spatial and temporal information. The batch of frames is used for video estimation. The number of frames in the batch is configurable and may be adjusted according to the type of microactivity to be monitored. Furthermore, the system is configured to detect at least one microactivity in the plurality of frames by a pre-trained AI model. The pre-trained AI model includes an object-hand association (OHA) module, a video classification module, an audio analysis (AA) module, a hand gesture (HG) module, a pose estimation (PE) module, an optical character recognition (OCR) module, a dimension measurement (DM) module, and an occupied hand (OH) module.The ODA module is based on deep learning (DL) methods involving neural networks, and the most important feature of DL is that it automatically extracts features to be focused on for analysis during the learning process. The output of each module is mapped to the exact duration of the microactivity performed. Furthermore, the system is configured to validate the microactivity detected by the pre-trained AI model, where the validation involves matching the detected microactivity with a ground truth sequence previously fed to the AI model. Condition checks are performed during the validation. For example, if the predicted result of the video classification module matches the ground truth list, no penalty is imposed and further activities are checked for correct tool usage and correct tool handling. However, if the predicted result of the video classification module does not match the ground truth list, a penalty is imposed and a certain predetermined score is deducted. Furthermore, the system is configured to assign a weight to each of the microactivity detected by the multiple modules, score the task by adding a positive score to each correctly performed microactivity, and assign a penalty to the incorrectly performed microactivity. The weighting assigned to each microactivity is based on correct tool usage and correct tool handling while performing the microactivity. Further, the system is configured to obtain a quality score for the task based on the individual scores assigned to the detected microactivities.
[0007] In yet another aspect, a computer program product is provided that includes a non-transitory computer-readable medium having a microactivity identification computer program integrated therein. The computer-readable program, when executed on a computing device, causes the computing device to receive a video stream via a plurality of synchronized video and audio camera devices via one or more hardware processors. The plurality of synchronized video and audio camera devices cover a task under surveillance from various angles. Furthermore, the computer-readable program, when executed on the computing device, causes the computing device to decode the video stream via one or more hardware processors by a capture device, which decodes the video stream into a plurality of frames and extracts associated audio from the video stream. A video stream of a person performing a task under surveillance is provided to the capture device for extracting a batch of frames from the video stream. The frame extraction results in the extraction of both spatial and temporal information. The batch of frames is used for video estimation. The number of frames in the batch is configurable and can be adjusted according to the type of microactivity monitored. Furthermore, the computer-readable program, when executed on a computing device, causes the computing device to detect at least one microactivity in the plurality of frames by a pre-trained AI model via one or more hardware processors. The pre-trained AI model includes an Object Hand Association (OHA) module, a Video Classification module, an Audio Analysis (AA) module, a Hand Gesture (HG) module, a Pose Estimation (PE) module, an Optical Character Recognition (OCR) module, a Dimension Measurement (DM) module, and an Occupied Hand (OH) module. The ODA module is based on the Deep Learning (DL) method involving neural networks, and the most important feature of DL is that it automatically extracts the features to be focused on for analysis during the learning process.The output of each module is mapped to the exact duration of the microactivity performed. The computer-readable program, when executed on a computing device, causes the computing device to verify the microactivity detected by the pre-trained AI model via one or more hardware processors, where the verification involves matching the detected microactivity with a ground truth sequence previously fed to the AI model. A condition check is performed during the verification. For example, if the predicted result of the video classification module matches the ground truth list, no penalty is imposed and further activities are checked for correct tool usage and correct tool handling. However, if the predicted result of the video classification module does not match the ground truth list, a penalty is imposed and a certain predetermined score is deducted. The computer-readable program, when executed on a computing device, causes the computing device to assign a weight to each of the microactivity detected by the multiple modules via one or more hardware processors, score the task by adding a positive score to each correctly performed microactivity, and assign a penalty to the incorrectly performed microactivity. The weighting assigned to each micro-activity is based on correct tool usage and correct tool handling while performing the micro-activity. The computer readable program, when executed on the computing device, causes the computing device, via one or more hardware processors, to obtain a quality score for the task based on the individual scores assigned to the detected micro-activities.
[0008] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is an example block diagram of a system for micro-activity identification, according to some embodiments of the present disclosure. [Diagram 2] FIG. 1 is an example diagram of a structural overview of a system for recording, identifying, and scoring micro-activity in surveillance video, according to some embodiments of the present disclosure. [Diagram 3] FIG. 2 is a flow diagram of an example method for recording, identifying, and scoring micro-activities engaged in a task under supervision, according to some embodiments of the present disclosure. [Figure 4] FIG. 2 illustrates a step of activity identification from a video classification module (M2) according to some embodiments of the present disclosure. [Diagram 5] FIG. 1 illustrates a generalized scheme for scoring multiple related micro-activities within a task under supervision, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates an implementation of a system for recording, identifying, and scoring micro-activities in surveillance video related to proper packaging task execution in a packaging industry setup, according to some embodiments of the present disclosure. [Figure 7] FIG. 13 is a diagram of a scorecard generated by the system for an example scenario of micro-activity identification in a packaging operation, according to some embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates an implementation of a system for recording, identifying, and scoring micro-activities in surveillance video of the performance of a laptop assembly task in an industrial setup, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Exemplary embodiments are described with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which that reference number first appears. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or similar parts. Examples and features of the disclosed principles are described herein, however, modifications, adaptations, and other implementations are possible without departing from the scope of the disclosed embodiments.
[0011] Many different techniques exist in the literature for human action recognition. Video-based action recognition is a new and challenging field of research, especially in this period, to identify and recognize actions in video sequences from surveillance streams. Action recognition in videos has many applications, such as content-based video retrieval, surveillance systems for security and privacy purposes, human-computer interaction, and activity recognition. The goal of action recognition is to detect and identify people, their behaviors, suspicious activities in the video, and deliver appropriate information to support interactive programs and IoT-based applications. Action recognition still poses many challenges when it comes to ensuring occupant security and safety, including industrial monitoring, violence detection, person identification, virtual reality, and cloud environments, due to significant improvements in camera movement, occlusions, complex backgrounds, and lighting changes. Spatial and temporal information play a key role in the recognition of different human actions in videos. In the last decade, the majority of methods used handcrafted feature engineering to indicate spatial attributes of dynamic motion to characterize the corresponding actions in the video. Handcrafted-based feature methods in action recognition are mainly database-oriented and cannot satisfy universal situations due to motion styles and complex background clutter.
[0012] Artificial intelligence (AI), especially deep learning (DL), is now a widely used technique for learning and creating high-level discriminative and salient features for end-to-end systems in video-based action and behavior recognition. Existing DL methods for human action recognition (HAR) use a simple convolutional neural network (CNN) strategy in convolution operations to learn features from video frames by using pre-trained models. These convolutional layers extract and learn spatial features to train models for classification. In comparison, traditional CNN models have lower performance than hand-crafted features in a sequence of data. Standard CNN models such as AlexNet, VGG, and ResNet learn spatial features from a single input image. Although these models are useful in capturing spatial information, they are less effective for temporal data, which is a key factor for capturing motion information for HAR in video sequences. Recently, the spatiotemporal problem has been addressed by using recurrent neural networks (RNNs), where LSTMs are designed to learn and process temporal features for HAR in surveillance systems, especially for long-term video sequences. At present, most of the researchers have developed two-stream approaches for action recognition to combine temporal and spatial features for joint feature training to cover the current challenges and limitations of HAR.
[0013] Based on these facts, accurate recognition of actions in real-world videos remains a challenging task, lacking information about motion, style, and background context for proper identification of human actions. Traditional methods are unable to address these issues due to the challenges in handling continuous actions, difficulties in modeling scenes crowded by occlusions, and sensitivity to noise. Similarly, recent methods for HAR solve the sequence learning problem by RNNs, LSTMs, and gated recurrent units. However, the above approaches do not focus on the detection of micro-activities related to the activity / task. The detection of micro-activities is very important for monitoring a person performing a task, and further scoring functions that can quantify the quality with which a micro-activity is performed by a person are important. Monitoring is important in all such activities that need to be monitored or under supervision.
[0014] In the following description, the terms “AI,” “machine learning (ML),” and “DL” are used interchangeably to refer to programs or algorithms that rely on training data to recognize patterns and make predictions or decisions.
[0015] In the following description, the terms "microactivity identification" and "microactivity recognition" are used interchangeably and both refer to programs or algorithms that rely on training data to recognize patterns and make predictions or decisions.
[0016] Referring now to the drawings, and more particularly to FIGS. 1-8, in which like reference characters indicate corresponding features consistently throughout the drawings, there are shown preferred embodiments which will be described in the context of the following exemplary systems and / or methods.
[0017] FIG. 1 illustrates an example block diagram of a system for micro-activity identification according to some embodiments of the present disclosure.
[0018] As shown in FIG. 1, the system 100 includes one or more processors 104, a communication interface device or input / output (I / O) interface 106, and one or more data storage devices or memories 102 operatively coupled to the one or more processors 104. The one or more processors 104, which are hardware processors, may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, graphic controllers, logic circuits, and / or any device that processes signals based on computational instructions. Among other capabilities, the processor is configured to fetch and execute computer-readable instructions stored in a memory. In the context of the present disclosure, the expressions "processor" and "hardware processor" may be used interchangeably. In one embodiment, the system 100 may be implemented in various computing systems, such as laptop computers, notebooks, handheld devices, workstations, mainframe computers, servers, network clouds, etc.
[0019] I / O interface 106 may include a variety of software and hardware interfaces, e.g., web interfaces, graphical user interfaces, etc., and may facilitate communications within a wide variety of networks and protocol types, including wired networks such as LAN, cable, wireless networks such as WLAN, cellular, or satellite, etc. In one embodiment, I / O interface 106 may include one or more ports for connecting several devices to each other or to another server.
[0020] The memory 104 may include any computer-readable medium known in the art, including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and / or non-volatile memory, such as read-only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk, and magnetic tape. In one embodiment, the memory 102 may include a database or repository. The memory 102 may include information regarding the input / output of the processor 104 of the system 100 and each step performed by the method of the present disclosure. In one embodiment, the database may be external (not shown) to the system 100 and coupled via the I / O interface 106. In one embodiment, the memory 102 includes multiple modules jointly responsible for microactivity identification. The multiple modules form an artificial intelligence (AI)-based pre-trained machine learning model 110. The multiple modules include an object-hand association (OHA) module M1, denoted as 110A, a video classification module M2, denoted as 110B, an audio analysis (AA) module M3, denoted as 110C, a hand gesture (HG) module M4, denoted as 110D, a pose estimation (PE) module M5, denoted as 110E, an optical character recognition (OCR) module M6, denoted as 110F, a dimension measurement (DM) module M7, denoted as 110G, and an occupied hand (OH) module M8, denoted as 110H. The ODA module 110A is based on a deep learning (DL) method involving neural networks, and the most important feature of DL is that it automatically extracts features to be focused on for analysis during the learning process. DL techniques used for object detection are classified as two-stage networks (RCNN and SPPNet, Mask R-CNN, Fast RCNN, Faster RCNN, Pyramid Networks / FPN, and G-RCNN) or one-stage networks (YOLO, SSD, RetinaNet, YOLOv3, YOLOv4, YOLOR).Convolutional Neural Networks (CNN) and You only look once (YOLO) are the most commonly applied DL models for object detection. The object detection module is trained by annotating every frame for object annotation by bounding box marking. The video classification 110B module tries to detect changes in activity classes for stationary activities. These modules need to be trained from beginning to end covering all activities so that the model can learn the sequence of micro-activities to be performed. Activity recognition is performed independent of position or orientation so that it is trained independent of the surrounding environment. The video classification module uses multi-modal sensor data from video and audio to classify the extent to which a micro-activity is involved in performing a task. The video classification module is trained on multiple cropped videos for each activity class. Spatiotemporal features are extracted using a convolutional neural network to obtain both spatial and temporal information for the activities. The video classification module recognizes the activities and their duration, and the video classification module classifies the activities in different classes. At least two activities with the highest probability classes are selected among the various classes identified by the video classification module. The selected activity is further verified using other modules configured according to the present invention to support the predictions made by the video classification module. The AA module 110C involves the process of converting, exploring, and interpreting audio signals recorded by digital devices. For the purpose of understanding the acoustic data, the acoustic data applies a wide range of techniques, including state-of-the-art deep learning algorithms. Audio analysis generally involves taking project-specific audio data stored in a standard file format and using software tools to prepare the data for your machine learning project.The audio analysis further includes extracting audio features from the visual representation of the acoustic data and selecting a machine learning model to train the machine learning model on the audio features. Audio analysis for acoustic event detection is performed via FFT performed on the audio extracted from the video file. Then, bandpass filters with lower and higher cutoff frequencies are tuned to the frequency of the acoustic event and used to extract the amplitude / energy of the signal. Whenever the spectral intensity / energy is significantly high (with respect to a set threshold), the acoustic event is detected. The lower and higher cutoff frequencies are obtained from a collection of audio clips carrying the target signal. The video stream with micro-activity associated with the acoustic event is detected in a video classification module and trimmed into small clips for audio detection. These trimmed small clips are used for audio detection. Audio is extracted from the video and a short-term FFT (SFFT) is calculated for amplitude comparison. The HG module 110D is integrated into the ML / DL model to recognize generally staged hand movements while performing an activity or task. The HG module is trained with the help of a dataset that contains images of a wide variety of hand gestures commonly used in the working environment. A machine learning algorithm is trained that can classify images of different hand gestures such as fist, palm, different target hand gestures like “pinching”, “thumbs up”, “pointing index finger”, “punch”, etc. Typically, hand gesture models are built using ConvNets (convolutional neural network) architectures to classify still images. The HG recognition model is used to detect correct tool handling and also to confirm the activities detected by the video classification model. Landmark points are detected for both right and left hands. These detected hand points are further classified using machine learning models to detect specific hand gestures like index finger, thumb right position.The pose estimation module 110E uses pose estimation methods involving computer vision techniques to detect human figures in images and videos, so that for example, where someone's elbow appears in the image can be determined. There is no personally identifiable information associated with pose detection, except for the estimation of major body joints. Body pose estimation uses deep learning algorithms for human pose detection. A PoseNet model is embedded to train the pose estimation module. PoseNet is a real-time pose detection technique where human poses can be detected in images or videos. PoseNet works in both single-mode (single person pose detection) and multi-pose detection (multiple person pose detection) cases. In simple terms, PoseNet is a deep learning TensorFlow model that allows estimating a person's pose by detecting body parts such as elbows, hips, wrists, knees, ankles, etc., and forming a skeletal structure of the person's pose by joining these points. The OCR module 110F is for text recognition. The OCR program extracts data from scanned documents, camera images, and image-only PDFs for further use. OCR software picks out characters on an image and replaces them with words, then replaces words with sentences, thus allowing the original content to be accessed and edited. OCR software also eliminates the need for manual data entry. CNN is one of the best techniques to use for deep learning OCR for the text detection step. Convolutional layers are usually used for image classification tasks due to their efficiency in feature extraction. Convolutional layers allow for the detection of meaningful edges in an image, as well as shapes and complex objects (at a higher level). In activity recognition, OCR is important because different parts of a machine may look the same, whereas the behavior of activities changes based on their location and number. The DM module 110G uses an image sensor for edge detection. In dimensional inspection using an image sensor, the position, width, and angle can be measured by capturing the object in two dimensions and detecting the edges.The dimensioning module is an ArUco™ marker, which is a binary square fiducial marker that can be used for camera pose estimation. The corner points of the marker are used for distance measurement by pose estimation and are the intersections of edges obtained by line fitting. The OHA module 110H maps the hand movement and detects the correct posture of the hand holding the tool to perform the assembly operation or any other task under supervision. The memory 102 further includes a database 112 for storing data related to the microactivity under supervision. The memory 102 further includes a number of modules (not shown) including programs or coded instructions that supplement the applications or functions executed by the system 100 to perform various steps involved in the process of microactivity identification. The modules can include, among others, routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. The modules can also be used as signal processors, node machines, logic circuits, and / or any other devices or components that process signals based on computational instructions. Further, the modules may be employed by hardware, by computer readable instructions executed by one or more hardware processors 104, or by a combination thereof. The modules may include various sub-modules (not shown).
[0021] FIG. 2 is an example diagram of a structural overview of a system for recording, identifying, and scoring micro-activity in surveillance video, according to some embodiments of the present disclosure.
[0022] FIG. 2 is an exemplary diagram of a structural overview of a system 100 for recording, identifying, and quality scoring micro-activities in surveillance videos, according to some embodiments of the present disclosure. As shown, a multi-camera video capture 202 receives a raw video stream from a video and audio capture device. The capture device serves as an interface for receiving and decoding the video stream to obtain a plurality of decoded frames from the video stream. The decoded frames are further calibrated to obtain calibrated frames. The portion of the video containing audio is directed to an audio extraction unit 204 where the audio is extracted, processed, and stored. The stored audio is used as input to an audio analysis module of a pre-trained AI model 110 according to the present invention. The AI model for micro-activity recognition 110, where the model 210 includes a series of modules denoted as M1, M2, ..., M8, performs individual tasks captured in the frames. In one embodiment of the present disclosure, the calibrated frames are processed in the AI module 110 via modules 110A-110H, each module identifying a specific activity associated with the micro-activity under surveillance. The system 100 includes a wrapper main() function 206 to determine micro-activities and quality scoring. The wrapper main() function 206 operates by detecting micro-activities identified as a result of the operation of one or more sub-modules (M1, M2, ..., M8) upon receiving input from a video and audio capture device and instructions from a micro-activity composition map tool 208. According to one embodiment of the present invention, the wrapper function performs feature selection by selecting the most significant and relevant micro-activities from a large set of a given data set. The system 100 includes a micro-activity composition map tool 208 that offers customization at the user end.The microactivity configuration map tool 208 contains all the predefined features, actions, and parameters to be used by the modules (110A-110H) to train the AI model 110. According to one embodiment of the present invention, the AI model 110 is directed by the microactivity configuration map tool 208, which coordinates between the submodules that detect microactivity. The user can define their own set of actions that are specific to the training of the modules to train them to make specific predictions. The video classification module classifies the microactivity into N classes, where N is the number of microactivities that need to be recognized. The microactivity configuration map 208 converges the subtasks captured by the multiple modules 210 (M1, M2, ..., M8) and processes the output of the AI model 210 together with the wrapper function 206. The binary square fiducial markers 212 (known as ArUco™ markers) are the DM module 110G. The DM module 110G first measures the coordinates of the captured object in the image frame by performing ArUco™ marking 212 on the image frame to calculate the pixel to inch conversion rate. Therefore, the DM module 110G calculates the object size based on the object coordinates and the pixel to inch conversion. The system 100 includes a microactivity validation tool 214. The microactivity validation tool 214 receives the output of the AI model 110 and a set of predefined instructions from the microactivity configuration map 208 and verifies the predictions made by the AI model 110. The output of each module is mapped to the exact duration of the microactivity to be executed.
[0023] The microactivity validation tool 214 is configured to perform key features such as sequence validation 214A, tool validation 214B, and presence / absence of microactivity 214C. Based on the detection of these key features, the validation tool 214 assigns a weighting to each microactivity. The weighting assigned to each microactivity is based on correct tool usage and correct tool handling while performing the microactivity. The weightings assigned to sequence validation 214A, tool validation 214B, and presence / absence of microactivity 214C are fed to a quality scoring module 216. A cumulative quality score is generated by the quality scoring module 216 for the detected activities via the pre-trained AI model 110 and is subsequently verified / verified by the validation tool 214. The quality scores thus generated are captured in a scorecard 218 and stored in a computer readable format. The scorecard 218 effectively captures complete details of the operator (e.g., the person performing the task / activity) involved in the activity, the activity class, and details of the multiple microactivities involved in the activity. The scorecard 218 generated as a result of the AI activity training provides real-time feedback prompts and assessments. Nevertheless, proper tool usage also ensures the safety of the operator performing the micro-activity.
[0024] FIG. 3 is a flow diagram of an example method for recording, identifying, and scoring micro-activities engaged in a task under supervision, according to some embodiments of the present disclosure.
[0025] The steps of the method 300 of the present disclosure will now be described with reference to the components or blocks of the system 100 as illustrated in FIGS. 1-8. Although process steps, method steps, techniques, and the like may be described in a sequential order, such processes, methods, and techniques may be configured to work in alternative orders. In other words, any sequence or order of steps described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any practical order. Additionally, some steps may be performed simultaneously.
[0026] In step 302 of the method 300, one or more hardware processors 104 are configured to receive a live video stream via multiple synchronized video and audio capture devices that together capture respective image views and can cover a field of view that focuses on micro-activities related to a task performed by a person. The received video signal includes an image stream of pixel data to be processed. The video and audio capture devices can be video cameras, monocular intensity cameras, stereo cameras, structured light cameras, Fairchild CCD cameras, time-of-flight (TOF) cameras, or any other camera, sensor, or system capable of capturing grayscale or color intensity images or depth images that are capable of accurate generation of video image signals at video rates. The camera / webcam typically does not have hardware to analyze the video it captures. Through offloading the analysis processing of the video to a processing unit, the video containing multiple micro-activities can be analyzed. In some embodiments, this analysis is performed in real-time or near real-time. In addition, the attached processing unit allows for automatic provisioning and flexible scaling of the infrastructure required to ingest and process streaming video data. Additionally, video streams from the same entity can be analyzed together (eg, tracking a person from camera to camera within a store).
[0027] In step 304 of the method 300, the one or more hardware processors 104 are configured to decode the video stream received from the video and audio capture device to obtain a plurality of frames. The extracted frames are further calibrated by removing distortion, and at the same time, audio from the video stream is extracted and processed to identify acoustic events related to the task. Audio extraction is performed for those frames with audio. The extracted audio is processed individually in the audio analysis module 204 of the system 100. The video received from the video and audio capture device is decoded to obtain frames. A capture device is provided that acts as an interface to receive and decode the video stream to obtain a plurality of decoded frames from the video stream. The videos captured from the multiple video capture devices are synchronized before decoding the video stream into frames. The synchronized video is decoded and the frames are extracted using multi-threading, and subsequently stored in a buffer for frame processing. The input frame rate can be about 20 FPS (frames per second) to about 60 FPS. In an exemplary implementation, the input frame rate is about 30 FPS. The size of the frames decoded from the live video stream is selected from 240p (pixels), 360p, 480p, 720p, 1080p, 1440p, and 2160p, with the rate of the size group including about 240p being about 30 FPS. The decoded frames are obtained for image calibration by applying at least one fiducial marker to remove distortion. The fiducial marker is used to correct image distortion while decoding the frame. A fiducial marker is an object placed in the field of view of the imaging system that appears in the image created to be used as a reference point or measure. The fiducial marker can be either one placed in or on the subject, or a mark or set of marks in the reticle of an optical instrument. The distortion is caused by the camera angle and needs to be corrected once by applying the fiducial marker.In one embodiment of the present disclosure, the fiducial marker is a binary square fiducial marker. The binary square fiducial marker provides the correspondence (its four corners) to obtain the camera pose. Also, the internal binary coding allows applying error detection and correction techniques. The binary square fiducial marker is an ArUco™ marker, a composite square marker composed of a wide black border, and an internal binary matrix that determines its identifier (id). The black border facilitates its fast detection in the image, and the binary coding allows its identification and the application of error detection and correction techniques. The marker size determines the size of the internal matrix. ArUco™ is calculated for the initial frame. The coordinates of the object are first determined, and then the identification of the ArUco™ marker is performed on the image to calculate the pixel to inch conversion rate. The object size is calculated from ArUco™ based on the object coordinates and the pixel to inch conversion.
[0028] In step 306 of the method 300, one or more hardware processors 104 are configured to process frames extracted from a video stream in a pre-trained AI model 110 for micro-activity identification, the model including a series of modules (M1, M2, ..., M8) performing individual tasks captured in said frames. During training of the AI model 110, a live video stream is captured via multiple synchronized video and audio capture devices. The video is decoded to obtain multiple frames. The extracted frames are subjected to calibration to remove distortion. At the same time, audio is extracted from the captured video for identification of any acoustic events related to the task. The decoded frames are stored in a storage medium. The frames include data in the form of audio, video, or images. The audio, video, or image data representing at least one task captured in the decoded frames is annotated to obtain a labeled dataset. This labeled dataset is then fed to an AI processor including modules (M1, M2, ..., M8) capable of identifying at least one task. The AI model is enabled to be trained based on the input data to identify at least one micro-activity in the image, audio, or video data. The trained AI model 110 is further used to perform task sequencing and micro-activity recognition. Thus, the pre-trained AI model 110 through multiple modules (M1, M2, ..., M8) identifies individual tasks captured in a frame, each of which contributes to specific object detection as well as human action recognition. A few modules, such as a video classification module, an audio analysis (AA) module, a hand gesture (HG) module, a pose estimation module, and an occupied hand module, are trained to recognize human actions, and a few modules, such as an object-hand association (OHA), an optical character recognition (OCR), and a dimensioning (DM), are trained to identify objects and activities.At least one microactivity is determined by combining the output of said modules with the extracted audio. The microactivity has been detected as a result of processing the audio and video input in the pre-trained AI model 110 and is guided by the configuration activity map tool to detect at least one microactivity. The configuration map tool coordinates between the modules of the AI model 110 that detect microactivity. The output of each module is mapped against the exact duration of the microactivity performed. The microactivity configuration map tool contains all the predefined features, actions, and parameters that should be used by the modules to train the AI model. The microactivity configuration map tool is customizable at the user end. The user can define their own set of actions that are specific to the training of the module to train it to make a specific prediction. The video classification module classifies the microactivity into N classes, where N is the number of microactivities that need to be recognized. To verify that the microactivity is correct, we use other modules such as hand gesture recognition, pose estimation, and occupied hand detection. The confirmed activities are used to check the ordering of the microactivity. The ODA module is used to detect the correct tool usage during the activity. HG Recognition is used to detect correct tool handling while performing an activity.
[0029] Each of the microactivities is a function of a module (SM) as described above, and the microactivity (μA i )teeth,
[0030]
number
[0031] where δ i, j= 0 or 1 based on Table-1 with the following configuration, which represents a sample activity map for four microactivities. The number 1 indicates that a particular microactivity has occurred and should be analyzed using a particular module, while the number 0 indicates that no activity representative of a particular model has occurred and therefore there is no contribution of a particular model. Based on the microactivity, a particular module is activated and the combination μA i represents a decision about a microactivity class based on a combination of module estimates.
[0032] [Table 1]
[0033] The switches are provided with an on-off mechanism that is triggered by the micro-activity configuration map, allowing specific modules to be activated based on input received from the camera / image capture device.
[0034] Upon receiving input from the camera / image capture device and instructions from the microactivity composition map tool, a wrapper function is provided that determines the microactivity detected as a result of the operation of one or more modules. The wrapper function performs feature selection by selecting the most significant and relevant microactivity from a large set of given datasets. The wrapper function is selected from any known wrapper method such as forward selection, backward elimination, and bidirectional elimination. The module that performs microactivity detection avoids specific activity training in all cases. This lends itself to a solution that is adaptable to new activities and does not have dependencies on large amounts of video, etc.
[0035] At step 308 of the method 300, the one or more hardware processors 104 are configured to validate the micro-activity based on a number of predefined conditions. The predefined conditions provide checkpoints for identifying specific micro-activity based on user-defined parameters within the system.
[0036] In step 310 of the method 300, the one or more hardware processors 104 are configured to assign a weight to each recognized microactivity and calculate a cumulative performance score for the person performing the task. The method provides an option to customize the weight assignment at the user end. A cumulative quality score is calculated for the activities detected via the AI model 110 and subsequently verified / validated. One or more microactivities detected and verified according to the process of the present invention are evaluated with a quality score that is meant to give a feel for how well the activity is performed by the person. A method for human activity recognition application in ubiquitous computing scenarios requires a focus on evaluating the quality of the activity, which goes beyond simply identifying the activity of interest. Objective quality assessment often requires domain-specific background information that is difficult to achieve and difficult to quantify, and generally biases the overall assessment and limits generalization. The framework for skill assessment in activity recognition enables automatic quality analysis of human activities. Based on the important features involved while performing the activities, the verification tool checks for sequence verification, tool verification, and the presence / absence of micro-activities, and such checking by the verification tool further involves assigning a weight to each micro-activity. The weight assigned to each micro-activity is based on the correct tool usage and correct tool handling while performing the micro-activity. The weight assignment is configurable at the user end. The weight may be set based on the importance identified by the user, and a quality score may be generated accordingly. A cumulative quality score is calculated for the activities detected via the AI model 110 and subsequently checked / verified by the verification tool. A penalty is deducted as soon as an incorrect tool usage is detected, an incorrect tool handling is detected, or an incorrect activity ordering is detected. The overall quality score is adversely affected by the penalty assignment.
[0037] The steps of the method for microactivity scoring are as follows: a) Receiving live video streams via multiple synchronized video and audio capture devices that together capture respective image views and can cover a field of view focusing on micro-activity related to the task to be performed by a human or by a machine pre-trained. b) decoding a video stream received from a video and audio capture device to obtain a number of frames, and calibrating the frames to remove distortions, while audio is extracted from the video stream and processed to identify acoustic events relevant to the task; c) The extracted frames are input for processing in a pre-trained AI model for micro-activity recognition, said AI model comprising a set of sub-modules performing distinct tasks captured in said frames. d) The output of said sub-modules and the extracted audio determine the identified micro-activity. e) The detected microactivity is checked against a number of predefined conditions. f) Weights are assigned to each identified micro-activity to derive a cumulative performance score for the person performing the task. g) The cumulative performance score is stored in a computer readable format.
[0038] In step 312 of the method 300, the one or more hardware processors 104 are configured to store the performance score in the form of a scorecard in a computer-readable format. The scorecard effectively captures complete details of the relevant operator (e.g., the person performing the task / activity), activity class, and details of multiple micro-activities within the activity. The scorecard further provides details on whether the micro-activity was performed correctly and what kind of anomaly was identified while performing the activity. The scorecard also captures the time within the video duration when the micro-activity (whether performed correctly or not) was detected. The scorecard generated as a result of the AI activity training provides real-time feedback prompts and assessments. In step 314 of the method 300, the one or more hardware processors 104 are configured to store the scorecard generated for the task. The scorecard is generated and stored in a computer-readable format. The scorecard effectively captures complete details of the relevant operator (e.g., the person performing the task / activity), activity class, and details of multiple micro-activities within the activity. Therefore, the AI model through multiple modules effectively detects the micro-activity. The method also encompasses video processing algorithms such as (a) tool zone mapping, (b) track and tag unit ID (screw id, etc.), (c) angle measurement between fingers to ensure safe gestures while holding the tool, along with AI models to make the method robust while identifying multiple micro-activities related to the task under supervision. The steps of scorecard generation are described as follows: a) Receiving live video streams via multiple synchronized video and audio capture devices that together capture respective image views and can cover a field of view that focuses on micro-activities related to the task being performed by a person. b) decoding a video stream received from a video and audio capture device to obtain a number of frames, and calibrating the frames to remove distortions, while audio is extracted from the video stream and processed to identify acoustic events relevant to the task; c) The extracted frames are input for processing in a pre-trained AI model for micro-activity recognition, said AI model comprising a set of sub-modules performing distinct tasks captured in said frames. d) The output of said sub-modules and the extracted audio determine the identified micro-activity. e) The detected microactivity is checked against a number of predefined conditions. f) Weights are assigned to each identified micro-activity to derive a cumulative performance score for the person performing the task. g) The cumulative performance score is stored in a computer readable format.
[0039] FIG. 4 illustrates steps of activity identification from the video classification module (M2) according to some embodiments of the present disclosure.
[0040] As shown, in step 402, the video classification module 110B receives a video stream of a person performing a task under supervision. In 404, a batch of frames is extracted from the video stream. The batch of frame extraction results in the extraction of both spatial and temporal information. The batch of frames is used for video estimation. The number of frames in a batch is configurable and can be adjusted according to the type of microactivity monitored. In 406, the extracted batch of frames is subjected to pre-processing. In pre-processing, a transformation is applied to the batch of frames to make the data suitable for estimation. The pre-processing results in a batch of processed frames. In 408, video estimation is performed on the batch of processed frames. The video estimation involves identifying multiple microactivities involved in the task. In 410, post-processing is applied to the individual estimates obtained by the multiple modules. The multiple modules predict at least one microactivity related to the task. Furthermore, the post-processing involves preparing an output of the video classification module 110B by identifying multiple microactivities related to the task captured in the batch of multiple frames.
[0041] FIG. 5 shows a generalized scheme for scoring multiple microactivities related in a task under supervision according to some embodiments of the present disclosure. The system 100 is configured to assign individual scores to each module related to microactivity recognition. Since the proposed method uses multiple modules for microactivity recognition, the output of all modules needs to be mapped with the correct activity at the correct duration. For each microactivity recognized by the respective module with respect to time, a log is generated. Hence, the log so generated has the mapped information for all modules. In step 502, the scoring starts by obtaining the mapped output logs of multiple microactivities captured by the individual modules (M1, N2, ..., M8). In step 504, the predicted activities of the video classification module are compared with the ground truth list to check the correct sequence of multiple microactivities. In step 506, the system 100 performs a condition check. If the predicted results of the video classification module 110B match with the ground truth list, the system 100 does not impose a penalty and further activities are checked for correct tool usage and correct tool handling. However, in the same step 506, if the predicted results of the video classification module 110B do not match the ground truth list, the system 100 imposes a penalty and a certain predefined score is deducted. The predefined score can be customized at the user end based on the sensitivity and importance of the micro-activity. Furthermore, in step 508, the system 100 imposes a penalty for not using the correct tool usage. Accordingly, a score is deducted for each incorrect usage of the tool. However, for each correct usage of the tool, a positive score is generated. Furthermore, in step 510, the system 100 imposes a penalty for not having the correct tool handling.Accordingly, a score is deducted for each incorrect tool handling. However, a positive score is generated for each correct tool handling. Finally, the system 100 generates an overall score by adding up all of the individual scores for correct tool use and correct tool handling, and by subtracting all penalties imposed on multiple modules and by subtracting incorrect tool use and incorrect tool handling.
[0042] Use case-I Micro-activity identification of packaging tasks in the packaging industry With reference to FIG. 6, a system and method for micro-activity recognition for proper packing task execution in a packaging industry setup is presented. An exemplary scenario illustrating the method of micro-activity recognition performed by the disclosed system 100 for packing tasks in a packaging industry setup is described below. Recognizing a packing task using standard supervised machine learning is challenging because observed data varies significantly depending on the number of items to be packed, the size of the items, and other parameters. This study focuses on characteristic and essential actions (motions) that occur in a specific task, such as stretching packing tape when assembling a shipping box, holding the tape, and a sequence of activities for packing a specific item. The micro-activities are performed by a human operator while performing the packing task and ensuring that the packing steps are performed properly.
[0043] Hardware setup: Two cameras (cam 1 and cam 2) are used to record the movements. Both cameras are equipped with microphones to record audio signals that can be used to verify some activity steps. The recording speed for capturing images was 30 FPS. The focus of the cameras was adjusted manually.
[0044] Critical steps are identified and required to be followed by the operator. 1. Prepare the packing space (μActivity 1) 2. Measure the product to be packed (μActivity 2) 3. Select the correct box for the product (μActivity 3) 4. Protect it by stuffing it in bubble wrap, air bags, etc. (μActivity 4) 5. Sealing the box (μActivity 5) 6. Sticking address labels (μActivity 6)
[0045] Identification of micro-activities is necessary to train the AI model. Describing in more detail the relevant micro-activities and their classification within a task / activity provides a better selection of sub-modules to make predictions. Based on each activity listed above, feature extraction is performed in step 604 by identifying the micro-activities involved while performing the packing task, sequence of activities, correct tool usage, and correct tool handling as listed in Table-2 below.
[0046] As shown in FIG. 6, the system 100 includes two cameras (cam 1 and cam 2) to capture activities. Both cameras are synchronized to capture a live video stream. In step 602, the input received from the first camera (cam 1) is subjected to frame extraction and calibration and sent to three sub-modules of the AI model. The frames extracted from the first camera are processed in a video classification module, an object hand association module (OHA), and an audio analysis module, followed by ArUco™ marking. From several activities performed and recorded at one time, activities with a confidence probability of at least two are captured as micro-activities to be identified. Furthermore, the activities identified from the AI model are sent for activity verification, where the activity verification tool applies several checkpoints to confirm the occurrence of the activities, including sequence verification as well. The AI model is presented with automatic activation of sub-modules based on the identified activities, for example, to detect the correct tool use, the object hand association module (OHA), and the audio analysis module activated with the ArUco™ marking. Similarly, in step 604, the input received from the second camera (cam 2) is subjected to frame extraction and calibration and sent to another three sub-modules of the AI model. The frames extracted from the second camera were processed in the hand gesture recognition module, pose estimation, and occupied hand detection. The AI model presents automatic activation of these three sub-modules based on the identified activity to detect correct tool handling, the object hand association module (OHA), and the occupied hand detection module used to detect correct tool use during activities such as pen detection, correct wrapping, correct address marking, sealing, labeling, etc., and known dimensional ArUco™ codes are placed to measure product size and box size.Further, a hand gesture and posture estimation module is used to detect correct tool handling to check hand movements during wrapping, number of wrappings done, double-checking of addressee, etc. Finally, an optical character recognition module is configured to check the placement of addressee. In step 606, for each activity listed above, an activity verification check is performed to check whether the correct tool is used and the tool is properly handled. Similarly, in step 608, the verified activities are subjected to one or more checks for sequence verification, and further scores are provided to the activity ordering. In step 610, the identified activities are captured from two different cameras, processed in optional sub-modules, and subjected to cumulative quality score calculation, and a scorecard is generated to provide real-time feedback. Based on the prediction made by the pre-trained AI model, the micro-activities involved in packing are identified and verified, and finally, a scorecard is generated by capturing the details of the operator performing the task as well as the presence / absence of the micro-activities along with the quality scores, and stored in the computer in CSV format or tabular form. The generated scorecard is shown in FIG. 7.
[0047] Quality Scoring Matrix: A cumulative quality score is generated by a quality scoring matrix for the detected activities via a pre-trained AI model including various modules. The quality scores thus generated are captured in a scorecard and stored in a computer-readable format. The quality scoring matrix offers configuration flexibility at the user end. The weighting of activities can be defined by the user based on their associated importance. An exemplary quality scoring matrix is presented as follows: 1. Presence and absence of subtasks. Weighting 30% (configurable) Score value[1] = (number of steps done incorrectly) / total number of steps (for example, if there are 12 steps and all steps are present, then score value[1] will be 12 / 12 = 1. But if 2 activities are missed, then score value[1] will be 10 / 12) 2. Sequence of subtasks, weighted 25 percent (configurable) Score value [2] = (number of incorrectly performed sequences) / total number of sequences 3. Correct tool use. Weighting 25 percent (configurable) Score value [3] = (number of tools used incorrectly) / total number of tools 4. Tool handling. Weighting 20 percent (configurable) Score value [4] = 1 I heard a click, 0 I didn't hear a click Final scoring: Sum of (score value[i] × weighting) / Sum of all weightings
[0048] In the scoring matrix, the sum of the scores varies between 0 and 1, if the activity is not performed correctly it is 0, if everything is done properly it is 1. The weighting given to each module is configurable. The sum of the weightings of all modules must equal 100. In this use case the weightings are taken into account as follows: ○ Video Classification: 40 ○ Correct Tool Use: 30 ○ Correct Tool Handling: 30 Total number of activities = 6 Total number of tools used = 11 Total number of activities where correct tool handling was checked = 4 o The following scenario presents the scores that would be generated if the user forgot to prepare the packing space and did not properly place the ruler for measurements.
[0049] [Table 2]
[0050] Therefore, the score for a packing task performed by an operator under supervision based on correct tool usage and correct tool handling would be 0.853. The scorecard gives a detailed analysis of the tasks performed.
[0051] Use case-II Verifying Activity Sequencing in Laptop Assembly with Correct Tool Use and Correct Tool Handling An exemplary scenario illustrating the method of micro-activity recognition performed by the disclosed system 100 for laptop assembly is described with reference to FIG. 8. For micro-activity identification during laptop assembly task, a previously trained AI model 110 is integrated to receive a video stream input of laptop assembly. The AI model is trained to identify micro-activities related to the laptop assembly task. The sequence of activities, correct tool usage, and correct tool handling are as listed in Table-3.
[0052] An operator performing a laptop assembly task is set under surveillance by capturing live video streams through a set of cameras installed to capture the activity. In step 802, the video streams captured through camera-1 and camera-2 are synchronized to process in a video capture device to obtain multiple frames of the scene. Videos from all cameras are captured. Since the number of frames and their occurrence time vary in case of a varying FPS (frames per second), synchronization of all videos is performed to maintain the same FPS. This leads to incorrect output since the activities performed do not overlap with the correct tool usage and correct tool handling. There are input buffers associated with all cameras. All buffers are input FPS sized and multi-threaded. In step 804, one thread of the synchronized video (thread 1) is processed as the first input buffer and is fed to the object hand association module (OHA) 110A and the video classification module 110B. Both 110A and 110B focus on the correct tool usage and estimate the correct tool usage based on the input thread of the synchronized video. Similarly, in step 806, another thread of synchronized video (thread 2) is processed as a second input buffer and fed to an object-hand association module (OHA) 110A and a video classification module 110B. Both 110A and 110B focus on correct tool usage and infer correct tool usage based on the input thread of synchronized video. This kind of approach of multi-camera identification of micro-activity presents close monitoring and accurate recognition via AI model 110. Video classification module 110B is used to classify spatio-temporal data. The model is trained using an efficient 3D network. A sliding window approach is used to prepare the data to be trained. In this approach, the input video is divided into one to several seconds according to activity classes. Furthermore, video transformations such as flipping, rotation, and jitter are applied to include various examples in the training dataset.The model is trained for 12 activity classes and videos are inferred every second with 30 seconds overlap. In step 808, hand gesture module 110D and pose estimation 110E are used to verify gestures performed during the activity. For this use case, the gestures used are tightening a screw, loosening a screw, holding a screwdriver, and holding the back cover of a laptop. Hand keypoints are detected using media pipes for both hands. Video streams captured from the first and second cameras are used for activity classification. Training data is prepared by extracting frames for specific hand gestures from the video. Hand keypoints are extracted for each gesture from multiple videos used for training. A customized deep learning model is used for classification of keypoints. Similarly, two different approaches are used to compute object-hand associations from both cameras. Hand-object interactions are identified using Interaction over Union (IOU). IOU is computed between bounding boxes of objects detected by Object-Hand Association Module (OHA). Since the hand is moving on multiple objects during the activity, false positives are also captured. This is further confirmed using a second camera by the occupying hand model 110H, which verifies if the hand is holding an object. The micro-activities mapped by modules 110D and 110H are presented in Table-4. The object hand association module 110A detects the items using the object hand association module (OHA). The detected objects for the laptop assembly task are screws, screwdriver, back cover, blower, hard disk, table, and RAM. YOLOv4 (one-stage object detector) is used for object detection. A score is generated based on the ground truth behavior and the actual behavior occurring while performing the micro-activity.Finally, zone-wise verification is performed from different sides of the object. The field of view from the top camera is divided into right and left zones with respect to the laptop. Using the Object Hand Association module (OHA), all objects are identified. Right and left zones are generated with respect to the laptop. The area of the zones is assigned using the bounding box coordinates of the nearest and furthest objects from each side. To check from which side of the table the object is picked up, we check from which side of the zone the object is missing during the activity. To identify this, the IOU between the zone and the object is calculated. The video classification module and the Object Hand Association module (OHA) are inferred in parallel. Based on the output of the video classification module, other modules like OCR (110F), audio (110C), and hand gestures (110D) are triggered. Now the activity ordering is verified by the video classification module. OHA (110H), OCR (110F), Audio Analysis (110C) and ArUco™ are used in validating correct tool usage, while hand gestures (110D) and pose estimation (110E) are used for correct tool handling. Since different modules use different camera angles based on viewpoint, the output of all modules is mapped in exact duration to verify that the activities are not only performed with correct tool usage and correct tool handling, but also in the correct sequence. The individual scores of all active modules used in the laptop assembly task are presented in Table-4.
[0053] [Table 3]
[0054] [Table 4A]
[0055] [Table 4B]
[0056] Scoring a laptop assembly task: The sum of the scores varies between 0 and 1. If the activity is not performed correctly it is 0, if everything is done properly it is 1. The weighting given to each module is configurable. The sum of all module weightings must equal 100. In this use case the weightings are assigned as follows: Video Classification: 30 Hand Gestures: 20 Hand object associations: 20 · Pose estimation: 20 Audio: 10
[0057] Based on the individual scoring of the video classification module, the hand gesture module, the occupant hand module, the pose estimation module, and the audio analysis module, the total score of the activity based on the system 100 is assigned as 0.86.
[0058] The microactivity identification proposed by system 100 is suitable for complex real-world situations where multiple microactivities occur simultaneously, such as monitoring people involved in the performance of an activity / task, monitoring robots, collaborative robots, humanoids, etc. involved in the performance of an activity / task.
[0059] The written description explains the subject matter herein to enable one of ordinary skill in the art to make and use the embodiments. The scope of the subject matter embodiments is defined by the claims, and may include other modifications that occur to those of ordinary skill in the art. Such other modifications are intended to fall within the scope of the claims if they have similar elements that do not differ from the literal language of the claims, or if they contain equivalent elements with minor differences from the literal language of the claims.
[0060] The embodiments of the present disclosure herein address the open problem of micro-activity identification, its effective capture, and quality scoring that provides real-time feedback of the event. Thus, the embodiments provide a quick solution that presents an effective way for an AI model to process inputs based on various types of micro-activity. Fiducial markings combined with a specific module that detects the dimensions of objects involved in the micro-activity provide accurate predictions for tool selection and handling. Scoring functions assigned to each detected micro-activity and configurable scoring matrix options provide an extremely reliable system where the operator's real-time feedback can be obtained along with the quality scores.
[0061] The scope of protection extends to such a program in addition to the computer readable means having therein the message, and such a computer readable storage medium contains program code means for the implementation of one or more steps of the method when the program runs on a server or mobile device or any suitable programmable device. The hardware device may be any kind of device that can be programmed, including any kind of computer, such as a server or personal computer, or any combination thereof. The device may also include hardware means, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a combination of hardware and software means, such as an ASIC and an FPGA, or means that may be at least one microprocessor and at least one memory in which a software processing component is placed. Thus, the means may include both hardware means and software means. The method embodiments described herein may be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments may be implemented on different hardware devices, for example using multiple CPUs.
[0062] The embodiments herein may include hardware and software elements. Software-implemented embodiments include, but are not limited to, firmware, resident software, microcode, etc. The functions performed by the various components described herein may be implemented with other components or combinations of other components. For purposes of this description, a computer-usable or computer-readable medium may be any apparatus that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.
[0063] The illustrated steps are intended to describe the illustrated exemplary embodiments, and ongoing technological developments should be expected to change the manner in which certain functions are performed. These examples are presented herein for illustrative purposes and are not limiting. Furthermore, boundaries of functional building blocks are arbitrarily defined herein for convenience of description. Alternative boundaries may be defined so long as the specified functions and their relationships are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc. described herein) will be apparent to those skilled in the art based on the teachings contained herein. Such alternatives fall within the scope of the disclosed embodiments. Similarly, the words "comprising," "having," "containing," and "including," as well as other similar formats, are intended to be equivalent in meaning, and that the items following any one of these words are open-ended in not implying an extensive list of such items or being limited to only the listed items. It should also be noted that, as used herein, in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.
[0064] Furthermore, one or more computer-readable storage media may be used to implement embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory in which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor to execute steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items, as well as to exclude carrier waves and transitory signals, i.e., non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage medium.
[0065] It is intended that the disclosure and examples be considered as exemplary only, with the true scope of the embodiments of the present disclosure being indicated by the following claims. [Explanation of symbols]
[0066] 100 Systems 102 Data storage device, memory 104 processors 106 Input / Output (I / O) Interface 108 Video and Audio Inputs 110 Pre-trained Machine Learning Models 110A Object Hand Association (OHA) Module M1 110B Video Classification Module M2 110C Audio Analysis (AA) Module M3 110D Hand Gesture (HG) Module M4 110E Attitude Estimation (PE) Module M5 110F Optical Character Recognition (OCR) Module M6 110G Dimension Measurement (DM) Module M7 110H Occupied Hand (OH) Module M8 112 Database 202 Multi-camera video capture 204 Audio Extraction Unit 206 Wrapper main() function 208 Microactivity Configuration Map Tool 210 Module 212 ArUco (trademark) marking 214 Microactivity Verification Tool 214A Sequence Verification 214B Tool Validation 214C Presence / Absence of Microactivity 216 Quality Scoring Module 218 Scorecard 300 ways
Claims
1. 1. A processor-implemented method (300) for scoring a task involving a plurality of microactivities, comprising: Receiving (302) video streams via a plurality of synchronized video and audio camera devices via one or more hardware processors; decoding (304) the video stream by a capture device via the one or more hardware processors, the capture device decoding the video stream into frames and extracting associated audio from the video stream; detecting (308) at least one microactivity in the plurality of frames by a pre-trained AI model via the one or more hardware processors; Validating (310) the micro-activity detected by the pre-trained AI model via the one or more hardware processors, where validation involves matching the detected micro-activity with ground truth sequences previously fed to the AI model; assigning a weight to each of the microactivities detected by a plurality of modules via the one or more hardware processors, scoring the task by adding a positive score to each correctly performed microactivity and assigning a penalty to each incorrectly performed microactivity (312); and obtaining (314) via the one or more hardware processors a quality score for the task based on individual scores assigned to the detected micro-activities.
2. 2. The method of claim 1, wherein the AI model includes an object-hand association module, a video classification module, an audio analysis module, a hand gesture module, a pose estimation module, an optical character recognition module, a dimension measurement module, and an occupying hand module.
3. The method of claim 1 , wherein the weighting assigned to each microactivity is based on correct tool usage and correct tool handling while performing the microactivity.
4. The method of claim 1 , wherein the output of each module is mapped to a precise duration of the microactivity that is executed.
5. A memory (102) for storing instructions; one or more communication interfaces (106); and one or more hardware processors (104) coupled to the memory (102) via the one or more communications interfaces (106), the one or more hardware processors (104) executing, in accordance with the instructions, receiving video streams via a plurality of synchronized video and audio camera devices; decoding the video stream by a capture device, the capture device decoding the video stream into a plurality of frames and extracting associated audio from the video stream; Detecting at least one microactivity in the plurality of frames with a pre-trained AI model; validating the micro-activities detected by the pre-trained AI model, where validation involves matching the detected micro-activities with ground truth sequences previously fed to the AI model; assigning a weight to each of the microactivities detected by a plurality of modules, scoring the task by adding a positive score to each correctly performed microactivity and assigning a penalty to each incorrectly performed microactivity; and deriving a quality score for the task based on individual scores assigned to the detected micro-activities.
6. 6. The system of claim 5, wherein the AI model includes an object-hand association module, a video classification module, an audio analysis module, a hand gesture module, a pose estimation module, an optical character recognition module, a dimension measurement module, and an occupying hand module.
7. The system of claim 5 , wherein the weighting assigned to each microactivity is based on correct tool usage and correct tool handling while performing the microactivity.
8. The system of claim 5 , wherein the output of each module is mapped to a precise duration of the microactivity that is executed.
9. One or more non-transitory machine-readable information storage media containing one or more instructions that, when executed by one or more hardware processors, receiving video streams via a plurality of synchronized video and audio camera devices; decoding the video stream by a capture device, the capture device decoding the video stream into a plurality of frames and extracting associated audio from the video stream; Detecting at least one microactivity in the plurality of frames with a pre-trained AI model; validating the micro-activities detected by the pre-trained AI model, where validation involves matching the detected micro-activities with ground truth sequences previously fed to the AI model; assigning a weight to each of the microactivities detected by a plurality of modules, scoring the task by adding a positive score to each correctly performed microactivity and assigning a penalty to each incorrectly performed microactivity; and obtaining a quality score for the task based on individual scores assigned to the detected micro-activities.
10. 10. The one or more non-transitory machine-readable information storage media of claim 9, wherein the AI model includes an object-hand association module, a video classification module, an audio analysis module, a hand gesture module, a pose estimation module, an optical character recognition module, a dimension measurement module, and an occupying hand module.
11. 10. The one or more non-transitory machine-readable information storage media of claim 9, wherein the weighting assigned to each microactivity is based on correct tool usage and correct tool handling while performing the microactivity.
12. 10. The one or more non-transitory machine-readable information storage media of claim 9, wherein the output of each module is mapped to a precise duration of the microactivity that is executed.
Citation Information
Patent Citations
Motion evaluation device, motion evaluation method, and motion evaluation system
JP2021071773A
