Factory data generation employing intelligent labeling

Abstract

The present specification presents a system and method for: receiving video data associated with at least one task executed by at least one worker; executing a human action recognition program to identify at least one action associated with the at least one task on the basis of the received video data; executing an object detection program to identify at least one object associated with the at least one task; identifying at least one sub-task of the at least one task on the basis of a combination of the identified at least one action and the identified at least one object associated with the at least one task; and generating at least first and second labels for each identified at least one sub-task on the basis of at least first and second labels associated, respectively, from a first set of existing labels for the identified action and from a second set of existing labels for the identified object.

Description

Factory Data Generation with Intelligent Labeling

[0001] The present disclosure relates generally to identifying specific subtasks within a larger task for worker training and / or human error identification.

[0002] In today's manufacturing environment, there is an increasing emphasis on worker training and human error identification. To achieve these goals, factories need to identify specific subtasks within larger tasks. While tasks are well-defined within manufacturing execution system (MES) software, subtask identification can prove difficult. One common approach for subtask identification is to divide the task into specific time intervals, but this approach may not work because each worker may perform the same subtasks at different intervals and in different orders. Therefore, there is a need to find a more sophisticated method of subtask identification that takes into account the variability in individual worker performance and order.

[0003] In some aspects, tasks may be specified in work orders from an MES system, but subtasks are associated with human activities and their movements. Figure 1 is a diagram 100 illustrating aspects of an industrial process including a product assembly task 120 associated with a component 110 and associated with fastening a screw 111. Product assembly task 120 may, in some aspects, be a task performed by a worker. Related subtasks may include picking up a screw 121, placing the screw on an assembly part 122, picking up a screwdriver 123, and tightening the screw 124, which may be repeated in the order shown in Figure 1 until all screws 111 are tightened. Figure 2 is a diagram 200 illustrating that related subtasks may be ordered differently by different workers or at different times. For example, diagram 200 shows that the tasks of picking up a screw 121 and placing the screw on the assembly 122 may be repeated until all screws 111 have been placed on the component 110, and then the tasks of picking up a screwdriver 123 and tightening the screw 124 may be repeated until all screws 111 have been tightened.

[0004] Detecting human-object relationships in video is an important challenge in action recognition. In this context, several methods utilize video frames to identify objects interacting with a person and classify the relationships between the person and the object. These methods propose a model called the Human-Object Relationship Transformer (HORT) to solve the above problem. The HORT model consists of two stages: still image prediction and feature extraction. These stages may be followed by intra- and / or inter-transformers that perform spatiotemporal inference. The HORT model integrates features from different instances across image planes and time steps, which can eliminate false-positive object proposals, identify active objects, and improve the accuracy of relationship classification.

[0005] In a factory environment, tasks are generally well-defined by the MES system, but segmenting and labeling the subtasks performed by workers can be a complex task. Dividing a task into specific time intervals is a common approach to identifying subtasks, but it may not always be effective because workers may perform subtasks at different intervals and in various orders, as shown in Figures 1 and 2. Therefore, a more sophisticated method of subtask identification that takes into account the variability in individual worker performance and order is needed.

[0006] The exemplary implementations described herein involve an innovative method for automatically identifying subtasks from video data associated with a task performed by one or more workers and generating labels associated with the identified subtasks and the video data. The method may include using a pre-trained human action recognition model using primitives and an object detection model using manufacturing-related objects to identify task change points (i.e., subtasks) from task video. The subtasks may be labeled with labels from a set of labels compiled from human action recognition and object detection. The labeled subtasks (or text associated with the labeled subtasks) may be summarized into a task description. The summarized task description may be evaluated by comparing it to the task description from the work order.

[0007] Aspects of the present disclosure include a method for receiving video data associated with at least one task performed by at least one worker, and executing a human action recognition program to identify at least one action associated with the at least one task based on the received video data. The method may further include executing an object detection program to identify at least one object associated with the at least one task based on the received video data, identifying at least one subtask of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task, and generating at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the at least one identified object from a second set of existing labels for the identified object, wherein the first set of existing labels and the second set of existing labels are stored in a memory.

[0008] Aspects of the present disclosure include a non-transitory computer-readable medium storing instructions for execution by a processor, the instructions may include instructions for executing a human action recognition program to receive video data associated with at least one task performed by at least one worker and identify at least one action associated with the at least one task based on the received video data, the instructions may further include instructions for executing an object detection program to identify at least one object associated with the at least one task based on the received video data, identifying at least one sub-task of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task, and generating at least one label for the identified at least one sub-task based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified action and at least a second label associated with the identified at least one object from a second set of existing labels for the identified object, wherein the first set of existing labels and the second set of existing labels are stored in a memory.

[0009] Aspects of the present disclosure include a system that may include means for receiving video data associated with at least one task performed by at least one worker and executing a human action recognition program to identify at least one action associated with the at least one task based on the received video data, the means for executing an object detection program to identify at least one object associated with the at least one task based on the received video data, identifying at least one subtask of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task, and generating at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the at least one identified object from a second set of existing labels for the identified object, the first set of existing labels and the second set of existing labels being stored in a memory.

[0010] Aspects of the present disclosure include an apparatus that may include a processor configured to receive video data associated with at least one task performed by at least one worker and execute a human action recognition program to identify at least one action associated with the at least one task based on the received video data, the processor may be further configured to execute an object detection program to identify at least one object associated with the at least one task based on the received video data, identify at least one subtask of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task, and generate at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the at least one identified object from a second set of existing labels for the identified object, the first set of existing labels and the second set of existing labels being stored in a memory.

[0011]

[0014] FIG. 1 illustrates an aspect of an industrial process including a product assembly task according to some aspects of the present disclosure.

[0015] FIG. 2 illustrates how related subtasks may be ordered differently by different workers or at different times.

[0016] FIG. 3 illustrates three work cells in a factory according to some aspects of the present disclosure.

[0017] FIG. 4 illustrates a clear overview of an architecture in which work orders are provided by an MES according to some aspects of the present disclosure.

[0018] FIG. 5 illustrates an example of task information tables sent from an MES system to respective edge analytics modules.

[0019] FIG. 6 illustrates a process of subtask labeling in which the edge analytics modules implement human activity recognition (HAR) and object detection programs.

[0020] FIG. 7 illustrates an example of subtask identification according to some aspects of the present disclosure.

[0021] FIG. 8 illustrates a flow diagram associated with a subtask identification module according to some aspects of the present disclosure.

[0022] FIG. 9 illustrates a flow diagram associated with task identification according to some aspects of the present disclosure.

[0023] FIG. 10 provides details regarding task description evaluation according to some aspects of the present disclosure.

[0024] FIG. 11 illustrates a flow diagram illustrating a method according to some aspects of the present disclosure.

[0025] FIG. 12 illustrates an example computing environment having an example computing device suitable for use in some exemplary implementations.

[0012] The following detailed description provides details of the figures and exemplary embodiments of the present application. Reference numbers and descriptions of elements that are duplicated between figures are omitted for clarity. Terms used throughout the description are provided by way of example and are not intended to be limiting. For example, use of the term "automatic" can include fully automatic or semi-automatic implementation, including user or administrator control over certain aspects of the implementation, depending on the desired implementation of one skilled in the art practicing the embodiments of the present application. Selection can be performed by a user via a user interface or other input means, or can be implemented via a desired algorithm. The exemplary embodiments as described herein can be used either alone or in combination, and the functionality of the exemplary embodiments can be implemented via any means depending on the desired implementation.

[0013] Worker training and human error identification are becoming increasingly important in modern manufacturing situations. To achieve these goals, factories need to identify specific subtasks within larger tasks. While tasks are clearly defined within MES software, identifying subtasks can be difficult. Dividing tasks into specific time intervals is a common approach to subtask identification, but it may not always be effective because workers may perform subtasks at different intervals and in various orders. Therefore, this novel approach provides a solution to automatically identify subtasks and provide appropriate labels for each subtask.

[0014] FIG. 3 is a diagram 300 illustrating three work cells (or workstations) in a factory, according to some aspects of the present disclosure. For example, in a first area, e.g., work cell 1, a first human worker 313 may be performing a manual task (such as assembly), and a camera 311 may capture the worker's movements and store the data on the edge module 310. Similarly, in a second work area, e.g., work cell 2, a second human worker 333 is working alongside a robotic arm 335 to complete a collaborative task. Again, camera 331 may capture video of the worker's movements and the robot's movements, and the data may be stored and processed on the edge module 330. Meanwhile, a third work area, e.g., work cell 3, may include a robotic arm 355 and may not include a human worker performing an automated task. Similar to the elements in the first and second work areas, the third work area may include a camera 351 capable of capturing the movements of the robotic arm 355, and the data may be stored and processed on the edge module 350. 3 thus provides an illustration of three scenarios, including two scenarios associated with the first two workspaces (i.e., Work Cell 1 and Work Cell 2) where there is a possibility of manual error based on the participation of a human worker, e.g., first human worker 313 or second human worker 333. FIG. 3 further illustrates a third scenario associated with the third workspace (i.e., Work Cell 3) where, for example, tasks have been automated, reducing the possibility of manual error (assuming the automation is properly configured).

[0015] FIG. 4 clearly illustrates an architectural overview in which work orders 401 are provided by MES 400. In some aspects, MES 400 may be a software-based system implemented by one or more computing devices (e.g., computing device 1305 in FIG. 13 , described below) used to manage and control production processes in a manufacturing environment. In some aspects, MES 400 may provide real-time information about the production process, allowing manufacturers to monitor and optimize production activities and make more educated decisions. MES typically include a wide range of functions, such as production scheduling, quality control, resource allocation, inventory management, and performance analysis. MES may help manufacturers reduce costs, increase efficiency, and improve quality. By providing real-time data and analytical results, manufacturers can identify areas where production can be improved and take corrective actions to optimize their operations. In some aspects, MES may help manufacturers observe and track products throughout the production process, ensuring that products meet quality standards and regulatory requirements.

[0016] In some aspects, MES 400 may be connected to a core analytics module 450, which collects data from a set of edge analytics modules (or from a set of edge analytics programs) present in or associated with each work cell (e.g., edge analytics module 410, edge analytics module 420, and edge analytics module 430). In some aspects, core analytics module 450 may process individual work orders 401 coming from MES 400. In each edge analytics module (edge ​​analytics modules 410, 420, and 430), a subtask labeling module (e.g., subtask labeling module 411, subtask labeling module 421, or subtask labeling module 431) can process video received from an associated recording device (e.g., one of cameras 311, 331, or 351, respectively) and provide an indication of a series of subtasks related to or comprising a task performed by a worker. A subtask labeling module (eg, subtask labeling module 411, 421, or 431) may further (automatically) generate a label for each of the subtasks.

[0017] The identified and labeled subtasks may be associated with the derived video to generate a set of labeled video data (e.g., metadata identifying the subtask, a timestamp associated with the subtask, and modified video data associated with the corresponding label). In some aspects, an edge analysis module (e.g., edge analysis module 410, 420, or 430) may provide the set of labeled video data to a connected labeled video database (DB) 440, which stores all of the videos received from the set of edge analysis modules (e.g., edge analysis module 410, 420, or 430) and the respective labels for each video.

[0018] FIG. 5 shows an example task information table sent from MES 400 to each edge analytics module. This table consists of product identification (ID) 510, work cell ID 520, and task information using task number field 530, task ID field 540, and task description field 550. For example, for product AAAA identified by product ID 510, a set of tasks may be specified in an ordered list identifying a first task 531 and a second task 532. The first task 531 (or second task 532) in the ordered list may be associated with a corresponding task ID 541 (or 542). Additionally or alternatively, a task description 551 (or 552) describing the type of task and what it entails or includes may also be associated with a particular task or task ID in some aspects. For example, task description 551 from a work order may be "Assemble part A onto part C by tightening screw 1." 5, or other data structures according to particular implementations, may in some aspects be generated by MES 400. MES 400 may in some aspects provide information regarding tasks performed in a particular workcell (e.g., the task ID and task description associated with the current task, or an ordered list of task IDs and task descriptions for a corresponding ordered list of tasks associated with a particular product ID 510) to a corresponding edge analytics module (e.g., one of edge analytics modules 410, 420, or 430 of FIG. 4).

[0019] FIG. 6 clearly illustrates the process of subtask labeling according to some aspects of the present disclosure. For example, edge analysis module 410, or one of edge analysis modules 420 or 430 of FIG. 4, in some aspects, may execute HAR program 610 (or include a HAR module) and execute object detection program 620 (or include an object detection module). HAR program 610, in some aspects, may be a pre-trained model capable of recognizing basic actions such as picking, placing, and reaching from video or video data. Object detection program 620, in some aspects, may be a pre-trained model capable of recognizing objects for a particular plant, task, or product from video or video data, and further capable of recognizing general objects such as screws, screwdrivers, hammers, or other common tools, fasteners, or parts. These two programs, HAR program 610 and object detection program 620, may be connected to and provide action results or object recognition to subtask identification program 630 (e.g., a program or module executed by or included in a subtask labeling module, e.g., one of subtask labeling modules 411, 421, or 431 of FIG. 4). In some aspects, subtasks may be identified using a set of recognized actions and an object associated with each action. For example, HAR program 610 may identify the action "pick up" associated with a particular set of video frames or timestamps, and object detection program 620 may identify an object that is a screwdriver in the same or overlapping set of video frames or timestamps, leading to identification of the subtask "pick up the screwdriver" by subtask identification program 630. In some aspects, subtask identification program 630 may identify change points and corresponding labels applied between each identified change point associated with the task video.In some aspects, the subtask identification program 630 can provide the identified change points and corresponding labels to the task identification process 640, where a summary of the task is generated from the subtask labels using natural language processing (NLP). Additionally, a task description is provided from the MES 400 through the work order 401 to each edge analysis module. The task description from the work order 401 can, in some aspects, be compared to the task description from the task identification process 640, and a similarity score can be generated by the task description evaluation module 650. If the similarity score is determined to be greater than a threshold similarity score at 660, the video analyzed by the HAR program 610, the object detection program 620, the subtask identification program 630, and the task identification process 640 can be labeled (e.g., associated with the change points and subtask labels) based on the video labeling module 680 and stored in the labeled video DB 440. If the similarity score is determined to be less than the threshold similarity score at 660, the subtask video and labels may be discarded at 670 without being stored in labeled video DB 440. In some aspects, the video may be provided to a human user, such as a subject matter expert, to label the subtasks to be used in subsequent training of the subtask labeling model.

[0020] FIG. 7 illustrates an example of subtask identification according to some aspects of the present disclosure. For example, subtask identification may be performed by one or more subtask identification modules, such as subtask labeling modules 411, 421, and / or 431 of FIG. 4 . In some aspects, this subtask identification may involve dividing a video clip into a (preconfigured) set of intervals, e.g., [T1, T5]. In some aspects, subtask identification may include identifying corresponding subtasks [S0001, S0002, S0001, S0002, S0003] based on transition points (e.g., boundaries between intervals) in the video clip. This type of method of dividing a video clip based on specific (e.g., preconfigured) time periods may be naive (e.g., a seemingly reasonable first-order approximation that fails in practical applications) because each worker may perform the same subtask at different time intervals (e.g., one worker may take 1 second, while another worker may take 1.5 seconds). Furthermore, the order of the subtasks may not be the same for each worker.

[0021] FIG. 7 further illustrates an exemplary subtask identification table that may be generated for a particular task 710 performed by a particular worker 715 in a particular work cell 720 after a subtask (e.g., subtask 761 or subtask 762) has been identified (e.g., by subtask ID 771 or subtask ID 772) and after a subtask description (or label) (e.g., subtask description 781 or subtask description 782) has been identified for or associated with that subtask.

[0022] FIG. 8 illustrates a flow diagram associated with the subtask identification program 630 according to some aspects of the present disclosure. For the purposes of the flow diagram illustrated in FIG. 8 , it is assumed that the human worker is primarily working with their hands, and accordingly, the associated determinations are related to wrist position and hand movement; however, other applications may use different indications of actions performed by the human worker. In some aspects, a module of the HAR program 810 may observe and analyze hand movement and monitor for a reaching action (e.g., an action in which the worker's hand reaches for a screwdriver) at 831 and determine that a reaching action was present. Once a reaching action is detected at 831, data regarding the wrist's feature point may be collected and / or determined at 832. Simultaneously, the object detection program 820 (e.g., corresponding to the object detection program 620) may also process objects in the vicinity of the wrist, and an object center may be calculated at 821. Thereafter, the distance between the collected and / or determined wrist feature point at 832 and the object center calculated at 821 may be calculated at 833. If the distance calculated at 833 is determined to be less than a threshold distance, epsilon (<ε), at 834, then in some aspects the process (e.g., subtask identifier 630) may continue by determining whether there has been a change in action at 835. If it is determined at 835 that the action has changed (e.g., from a reach action to a pick up action), a subtask change point may be identified at 836. Based on this identification, text associated with the object may be stored in text DB 841 and text associated with the action may be stored in label DB 842 (e.g., along with a common identifier or some other indication of the correspondence between the data stored in text DB 841 and label DB 842). However, if the distance calculated at 833 is determined to be greater than epsilon at 834, then in some aspects it may be determined whether a release action has been detected at 837. If a release action is detected at 837, then the task is identified as completed at 838.If either an action is determined to have changed at 835 or a subtask change point is identified at 836, the distance between the wrist pivot point collected and / or determined at 832 and the object center calculated at 821 can be calculated at 833 (e.g., for a subsequent time) and the process may continue to completion at 838 (e.g., a release action is determined to have been detected at 837).

[0023] 9 shows a flow diagram associated with the task identification process 640 according to some aspects of the present disclosure. For example, text from the text DB 841 associated with the subtask identification program 930 (e.g., corresponding to the subtask identification program 630) may be passed through a named entity recognition module at 941 (as used herein, a module may refer to a program, or a program or process, e.g., a subprogram associated with the task identification process 640) so that key entities may be identified and extracted from the text, such as object names, actions, or other entities / labels as needed for the particular analysis task. A text clustering module may then process the text at 942 based on a trained model such that text with similar ideas or concepts is grouped together. After processing by the text clustering module at 942, the text may be processed by a text summarization algorithm at 943. The text summarization algorithm applied at 943 may, in some aspects, be an abstract or abstract summarization algorithm to generate summaries from the grouped text resulting from processing by the text clustering module at 942. The summarized text may be expected to explain the overall task based on its subtasks and may be used to evaluate how well the subtask identification and labeling was performed. After the text is summarized by the text summarization algorithm at 943, a text evaluation algorithm may be used at 944 to evaluate the quality of the summary. For example, in some aspects, the text evaluation algorithm may be implemented at 944 by checking the quality of the summary using Recall-Oriented Understudy for Gisting Evaluations (ROUGE) metrics. In some aspects, the summary may be refined by a refinement process at 945 by adjusting parameters, optimizing the algorithm, etc. After refinement at 945, the summarized text may be output at 946 for further evaluation.

[0024] 10 provides details regarding task description evaluation module 650, according to some aspects of the present disclosure. In some aspects, a task description (e.g., sentence A 1011) from work order 1010 (e.g., corresponding to work order 401) and a task description (e.g., sentence B 1051) from task identification module 1040 (e.g., corresponding to task identification process 640) are passed through a bidirectional encoder representation from transformer (BERT) model 1012 and a BERT model 1052, respectively. In some aspects, BERT provides embeddings for all of the words in the sentence. Those embeddings from the BERT model are then passed through a pooling operation (e.g., for sentence A 1011 and sentence B 1051 in 1013 and 1053, respectively). For example, the pooling at 1013 and / or 1053 may include averaging all of the terms, or some other suitable processing. After the pooling at 1013 and / or 1053, in some aspects, a single vector representation u may be generated at 1014 and a single vector representation v may be generated at 1054 for each sentence, respectively. Similarities may be calculated between the single vectors (e.g., u and v) at 1055. For example, cosine similarity may be calculated between two single vectors (e.g., u and v) at 1055, and the similarity score may be stored at 1056.

[0025] 11 is a flow diagram 1100 illustrating a method according to some aspects of the present disclosure. In some aspects, the method is performed by an edge device (e.g., edge modules 310, 330, 350, edge analytics modules 410, 420, 430, or computing device 1305) that may be part of a system that also includes a video recording device (e.g., camera 311, 331, 351) and / or an analytics device (e.g., core analytics module 450 or computing device 1305) that may perform various analyses of the recorded data, store automatically labeled recorded data, and present the results of the analyses for display to a user. For example, the video recording device may record video data associated with at least one task performed by at least one worker. The at least one task may, in some aspects, be a predefined task associated with an industrial process. Each task of the at least one task may, in some aspects, be associated with a corresponding set of subtasks. In some aspects, as described above with respect to at least FIGS. 1 and 2, the corresponding sets of subtasks may be different (e.g., ordered differently) for each task, even if the tasks are associated with the same work order (e.g., product assembly).

[0026] A video recording device, in some aspects, may provide an edge device with video data associated with at least one task performed by at least one worker. For example, a camera 311 may provide video data (e.g., video data 701) to an edge module 310. With reference to FIG. 13 , for example, an input device such as a camera may provide video data via a user interface 1335 (or via a network 1350 and / or an I / O interface 1325) to a computing device 1305 associated with (or implementing) the edge module.

[0027] At 1121, a device (e.g., an edge device) may receive video data associated with at least one task performed by at least one worker from a video recording device. For example, edge module 310 may receive video data (e.g., video data 701) from camera 311. With reference to FIG. 13 , for example, computing device 1305 associated with (or implementing) an edge module (or edge device) may receive video data from an associated input device, such as a camera, via user interface 1335 (or via network 1350 and / or I / O interface 1325).

[0028] At 1122, the device (e.g., edge device) may execute a human action recognition program to identify at least one action associated with the at least one task based on the received video data. For example, the human action recognition program may be executed by, for example, processor 1310 (e.g., alone or in combination) implementing and / or including HAR program 610 or 810 or edge analytics module 410 (or 420 or 430). In some aspects, the identified at least one action may be associated with at least one of the at least one worker's hand or the at least one worker's wrist. The identified at least one action may, in some aspects, include one or more of: reaching for an object, picking up an object, releasing an object, pushing an object, lifting an object, pulling an object, carrying an object, or twisting an object (examples of possible actions; other actions may also be identified based on the particular task being performed by the human worker).

[0029] At 1123, the device (e.g., edge device) may execute an object detection program to identify at least one object associated with the at least one task based on the received video data. For example, the object detection program may be executed by processor 1310 (e.g., alone or in combination) or edge analytics module 410 (or 420 or 430), which implements and / or includes object detection program 620 or 820, for example. In some aspects, the object may be one of a tool (e.g., a screwdriver, hammer, wrench, etc.), a fastener (e.g., a screw, dowel, nut / bolt, etc.), or a part (a panel, a housing, etc.) associated with the at least one task.

[0030] At 1124, the device (e.g., edge device) may identify at least one subtask of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task. For example, identifying the at least one subtask of the at least one task may be performed, for example, by processor 1310 (e.g., alone or in combination) or edge analysis module 410 (or 420 or 430) that includes and / or implements subtask identification program 630. In some aspects, to identify the at least one subtask of the at least one task, the device (e.g., edge device) may be configured to identify one of a start or an end of the at least one subtask based on a distance of at least one of at least one worker's hand or at least one worker's wrist from the at least one identified object.

[0031] At 1125, the device (e.g., edge device) may generate at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the at least one identified object from a second set of existing labels for the identified objects. For example, generating the at least one label may be performed by, for example, subtask labeling module 411 (or 421 or 431) and / or processor 1310 (e.g., alone or in combination) or edge analysis module 410 (or 420 or 430) implementing task identification process 640. In some aspects, the first set of existing labels and the second set of existing labels are stored in a memory associated with the device (e.g., memory 1315, external storage 1345, or text DB 841).

[0032] In some aspects, the device (e.g., an edge device) may calculate a similarity score between the generated at least one label and the task description provided for the predefined task. For example, calculating the similarity score may be performed, sentence by sentence, by, for example, a processor 1310 (e.g., alone or in combination) including and / or implementing task description evaluation module 650 or edge analysis module 410 (or 420 or 430), a BERT model applied in task identification module 1040, 1012 and / or 1052, pooling in 1013 and / or 1053, single vector representation generation in 1014 and / or 1054, calculation of cosine similarity between the two single vectors in 1055, and storage of the similarity score in 1056. This similarity score may, in some aspects, be calculated using any language model capable of evaluating the similarity of phrases and / or natural language descriptions of tasks or subtasks.

[0033] In some aspects, the device (e.g., edge device) may determine whether to store or discard the generated at least one label for further analysis based on the calculated similarity score. For example, this determination may be performed by processor 1310 (e.g., alone or in combination) or edge analytics module 410 (or 420 or 430), which includes and / or performs the determination at 660. In some aspects, this determination may include determining whether the similarity value is above or below a threshold.

[0034] In some aspects, a device (e.g., an edge device) or an associated device (e.g., an analysis device) may analyze the generated set of labels for subtasks of at least one task for each of multiple workers performing at least one task to determine an optimal subtask ordering associated with performing the at least one task. With reference to FIG. 13 , for example, an additional computing device 1305 associated with (or implementing) the analysis module (or analysis device), e.g., a set of additional processors 1310 (e.g., alone or in combination) including and / or implementing edge analysis module 410 (or 420 or 430) or core analysis module 450, may analyze the generated set of labels. In some aspects, this analysis may compare information associated with different workers or different analyzed videos for a particular task (e.g., for a particular task 710) to determine an ordering of subtasks (e.g., subtasks 761 or 762) associated with a minimum average time, a maximum number of parts produced per shift, or some other parameter used to optimize the particular task.

[0035] In some aspects, a device (e.g., an edge device) or an associated device (e.g., an analysis device) may generate and store modified video data associated with at least one task performed by at least one worker. For example, generating and storing the modified video data may be performed by processor 1310 (e.g., alone or in combination) that includes and / or implements edge analytics module 410 (or 420 or 430) or core analytics module 450. This modified video data, in some aspects, may include received video data and metadata indicating the generated at least one label and a timestamp associated with one or more of the start of the identified at least one subtask, the end of the identified at least one subtask, or the duration of the identified at least one subtask. In some aspects, the analysis device may analyze a set of generated labels for subtasks of the at least one task for each of multiple workers performing the at least one task by analyzing the modified video data. Storing the modified video data, in some aspects, may include providing the modified video data to a common memory associated with the analysis device.

[0036] In some aspects, a device (e.g., an edge device) or an associated device (e.g., an analysis device) may present a visual representation of the optimal subtask ordering via a display. For example, this presentation may be performed by a processor 1310 (e.g., alone or in combination) that includes and / or implements edge analysis module 410 (or 420 or 430) or core analysis module 450 via I / O interface 1325 and / or output device / interface 1340.

[0037] 12 is a flow diagram 1200 illustrating a method according to some aspects of the present disclosure. In some aspects, the method is performed by a system including a video recording device 1210 (e.g., camera 311, 331, 351), an edge device 1220 (e.g., edge module 310, 330, 350, edge analytics module 410, 420, 430, or computing device 1305), and / or an analytics device 1230 (e.g., core analytics module 450 or computing device 1305) that can perform various analyses of the recorded data, store automatically labeled recorded data, and present the results of the analyses for display to a user. At 1211, the device (e.g., video recording device 1210) can record video data associated with at least one task performed by at least one worker. The at least one task may, in some aspects, be a predefined task associated with an industrial process. Each task of the at least one task may, in some aspects, be associated with a corresponding set of subtasks. In some aspects, as described above with respect to at least FIGS. 1 and 2, the corresponding set of subtasks may be different (e.g., ordered differently) for each task, even if the tasks are associated with the same work order (e.g., product assembly).

[0038] At 1212, a device (e.g., video recording device 1210) may provide video data associated with at least one task performed by at least one worker to edge device 1220. For example, camera 311 may provide video data (e.g., video data 701) to edge module 310. Referring to FIG. 13 , for example, an input device such as a camera may provide video data via user interface 1335 (or via network 1350 and / or I / O interface 1325) to computing device 1305 associated with (or implementing) the edge module.

[0039] At 1221, a device (e.g., edge device 1220) may receive video data associated with at least one task performed by at least one worker from video recording device 1210. For example, edge module 310 may receive video data (e.g., video data 701) from camera 311. With reference to FIG. 13 , for example, computing device 1305 associated with (or implementing) an edge module (or edge device) may receive video data from an associated input device, such as a camera, via user interface 1335 (or via network 1350 and / or I / O interface 1325).

[0040] At 1222, the device (e.g., edge device 1220) may execute a human action recognition program to identify at least one action associated with the at least one task based on the received video data. For example, the human action recognition program may be executed by, for example, processor 1310 (e.g., alone or in combination) implementing and / or including HAR program 610 or 810 or edge analytics module 410 (or 420 or 430). In some aspects, the identified at least one action may be associated with at least one of the at least one worker's hand or the at least one worker's wrist. The identified at least one action may, in some aspects, include one or more of: reaching for an object, picking up an object, releasing an object, pushing an object, or twisting an object (examples of possible actions; other actions may also be identified based on the particular task being performed by the human worker).

[0041] At 1223, a device (e.g., edge device 1220) may execute an object detection program to identify at least one object associated with the at least one task based on the received video data. For example, the object detection program may be executed by processor 1310 (e.g., alone or in combination) or edge analytics module 410 (or 420 or 430), which implements and / or includes object detection program 620 or 820, for example. In some aspects, the object may be one of a tool (e.g., a screwdriver, hammer, wrench, etc.), a fastener (e.g., a screw, dowel, nut / bolt, etc.), or a part (a panel, a housing, etc.) associated with the at least one task.

[0042] At 1224, the device (e.g., edge device 1220) may identify at least one subtask of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task. For example, identifying the at least one subtask of the at least one task may be performed, for example, by processor 1310 (e.g., alone or in combination) or edge analysis module 410 (or 420 or 430) that includes and / or implements subtask identification program 630. In some aspects, to identify the at least one subtask of the at least one task, the device (e.g., edge device 1220) may be configured to identify one of a start or an end of the at least one subtask based on a distance of at least one of at least one worker's hand or at least one worker's wrist from the at least one identified object.

[0043] At 1225, the device (e.g., edge device 1220) may generate at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the identified at least one object from a second set of existing labels for the identified objects. For example, generating the at least one label may be performed by, for example, subtask labeling module 411 (or 421 or 431) and / or processor 1310 (e.g., alone or in combination) or edge analysis module 410 (or 420 or 430) implementing task identification process 640. In some aspects, the first set of existing labels and the second set of existing labels are stored in a memory associated with the device (e.g., memory 1315, external storage 1345, or text DB 841).

[0044] At 1226, the device (e.g., edge device 1220) may calculate a similarity score between the generated at least one label and the task description provided for the predefined task. For example, calculating the similarity score may be performed, sentence by sentence, by, for example, a processor 1310 that includes and / or implements task description evaluation module 650 (e.g., alone or in combination) or edge analysis module 410 (or 420 or 430), task identification module 1040, BERT model 1012 or 1052, pooling 1013 or 1053, single vector representation 1014 and 1054, cosine similarity calculation 1055 between two single vectors, and similarity score storage 1056. This similarity score may, in some aspects, be calculated using any language model capable of evaluating the similarity of phrases and / or natural language descriptions of tasks or subtasks.

[0045] At 1227, the device (e.g., edge device 1220) may determine whether to store or discard the generated at least one label for further analysis based on the calculated similarity score. For example, this determination may be performed by processor 1310 (e.g., alone or in combination) or edge analytics module 410 (or 420 or 430), which includes and / or performs the determination at 660. In some aspects, this determination may include determining whether the similarity value is above or below a threshold.

[0046] At 1231, an apparatus (e.g., analysis apparatus 1230) may analyze the generated set of labels for subtasks of the at least one task for each of a plurality of workers performing the at least one task to determine an optimal subtask ordering associated with performing the at least one task. With reference to FIG. 13 , for example, an additional computing device 1305 associated with (or implementing) the analysis module (or analysis apparatus), e.g., a set of additional processors 1310 (e.g., alone or in combination) including and / or implementing edge analysis module 410 (or 420 or 430) or core analysis module 450, may analyze the generated set of labels. In some aspects, this analysis may compare information associated with different workers or different analyzed videos for a particular task (e.g., for a particular task 710) to determine an ordering of subtasks (e.g., subtasks 761 or 762) associated with a minimum average time, a maximum number of parts produced per shift, or some other parameter used to optimize the particular task.

[0047] At 1232, a device (e.g., analysis device 1230) may generate and store modified video data associated with at least one task performed by at least one worker. For example, generating and storing the modified video data may be performed by processor 1310 (e.g., alone or in combination) that includes and / or implements edge analysis module 410 (or 420 or 430) or core analysis module 450. This modified video data may, in some aspects, include the received video data and metadata indicating the generated at least one label and a timestamp associated with one or more of the start of the identified at least one subtask, the end of the identified at least one subtask, or the duration of the identified at least one subtask. In some aspects, the analysis device may analyze a set of generated labels for subtasks of the at least one task for each of multiple workers performing the at least one task by analyzing the modified video data. Storing the modified video data at 1232 may, in some aspects, include providing the modified video data to a common memory associated with the analysis device.

[0048] At 1233, the device (e.g., analysis device 1230) may present a visual representation of the optimal subtask ordering via a display. For example, this presentation may be performed by a processor 1310 (e.g., alone or in combination) that includes and / or implements edge analysis module 410 (or 420 or 430) or core analysis module 450 via I / O interface 1325 and / or output device / interface 1340.

[0049] Worker training and human error identification are becoming increasingly important in modern manufacturing situations. To achieve these objectives, factories need to identify specific subtasks within larger tasks. While tasks are clearly defined within MES software, identifying subtasks can be difficult. Dividing tasks into specific time intervals is a common approach to subtask identification, but it may not always be effective because workers may perform subtasks at different intervals and in various orders. Therefore, this novel approach provides a solution to automatically identify subtasks and provide appropriate labels for each subtask.

[0050] 13 illustrates an exemplary computing environment having an exemplary computing device suitable for use in some exemplary implementations. The computing device 1305 in the computing environment 1300 can include one or more processing units, cores, or processors 1310, memory 1315 (e.g., RAM, ROM, and / or the like), internal storage 1320 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interface 1325, any of which can be coupled over a communication mechanism or bus 1330 for communicating information or can be incorporated into the computing device 1305. The I / O interface 1325 can be further configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0051] The computing device 1305 may be communicatively coupled to an input / user interface 1335 and an output device / interface 1340. Either or both of the input / user interface 1335 and the output device / interface 1340 may be a wired or wireless interface and may be detachable. The input / user interface 1335 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, a touchscreen interface, a keyboard, pointing / cursor control, a microphone, a camera, Braille, a motion sensor, an accelerometer, an optical reader, and / or the like). The output device / interface 1340 may include a display, television, monitor, printer, speakers, Braille, or the like. In some exemplary implementations, the input / user interface 1335 and the output device / interface 1340 may be incorporated with or physically coupled to the computing device 1305. In other exemplary implementations, other computing devices may function as or provide the functionality of input / user interface 1335 and output device / interface 1340 for computing device 1305 .

[0052] Examples of computing devices 1305 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices mounted on vehicles and other machines, devices carried by people and animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded and / or televisions with one or more processors coupled thereto, radios, and the like).

[0053] Computing device 1305 may be communicatively coupled (e.g., via I / O interface 1325) to external storage 1345 and to a network 1350 for communication with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1305 or any connected computing device may function as, provide services to, or be referred to as a server, client, thin server, general-purpose machine, special-purpose machine, or other label.

[0054] I / O interface 1325 may include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, and the like) for communicating information to and / or from at least all connected components, devices, and networks in computing environment 1300. Network 1350 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, and the like).

[0055] The computing device 1305 may use and / or communicate using computer-usable or computer-readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, fiber optics), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tape), optical media (e.g., CD-ROMs, digital video disks, Blu-ray discs), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory ("Blu-ray" is a registered trademark).

[0056] The computing device 1305 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions can be retrieved from transitory media and stored on and retrieved from non-transitory media. The executable instructions can be from one or more of any programming, scripting, and machine language, such as C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others ("Java" and "JavaScript" are registered trademarks).

[0057] The processor 1310 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1360, an application programming interface (API) unit 1365, an input unit 1370, an output unit 1375, and an inter-unit communication mechanism 1395 for different units to communicate with each other, with the OS, and with other applications (not shown). The above-mentioned units and elements can vary in design, function, configuration, or implementation and are not limited to the above description. The processor 1310 can have the form of a hardware processor, such as a central processing unit (CPU), or can be a combination of hardware and software units.

[0058] In some exemplary implementations, when information or instructions for execution are received by API unit 1365, it may be communicated to one or more other units (e.g., logic unit 1360, input unit 1370, output unit 1375). In some examples, logic unit 1360 may be configured to control the flow of information between units and, in some exemplary implementations described above, direct the services provided by API unit 1365, input unit 1370, and output unit 1375. For example, the flow of one or more processes or implementations may be controlled by logic unit 1360 alone or in conjunction with API unit 1365. Input unit 1370 may be configured to obtain inputs for the calculations described in the exemplary implementations, and output unit 1375 may be configured to provide outputs based on the calculations described in the exemplary implementations.

[0059] The processor 1310 may be configured to receive video data associated with at least one task performed by at least one worker. The processor 1310 may be configured to execute a human action recognition program to identify at least one action associated with the at least one task based on the received video data. The processor 1310 may be configured to execute an object detection program to identify at least one object associated with the at least one task based on the received video data. The processor 1310 may be configured to identify at least one subtask of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task. In some aspects, the processor 1310 may also be configured to identify one of a start or an end of the at least one subtask based on a distance of at least one of the at least one worker's hand or the at least one worker's wrist from the at least one identified object. The processor 1310 may be configured to generate at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the at least one identified object from a second set of existing labels for the identified objects. In some aspects, the processor 1310 may also be configured to calculate a similarity score between the generated at least one label and a task description provided for the predefined task. In some aspects, the processor 1310 may also be configured to determine whether to store the generated at least one label for further analysis or discard it based on the calculated similarity score.

[0060] In some aspects, for example, when computing device 1305 is associated with a video recording device, processor 1310 may be configured to record video data associated with at least one task performed by at least one worker. In some aspects, processor 1310 may also be configured to provide, to an edge device, the video data associated with at least one task performed by at least one worker.

[0061] For the computing device 1305 associated with the analysis device or edge device, in some aspects, the processor 1310 may be configured to analyze the set of generated labels for the subtasks of the at least one task for each of a plurality of workers performing the at least one task to determine an optimal subtask ordering associated with performing the at least one task. The processor 1310 may also be configured to present a visual representation of the optimal subtask ordering via a display. The processor 1310 may also be configured to generate and store modified video data associated with the at least one task performed by the at least one worker. The processor 1310 may also be configured to analyze the set of generated labels for the subtasks of the at least one task for each of a plurality of workers performing the at least one task by analyzing the modified video data.

[0062] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In exemplary implementations, the performed steps require physical manipulations of tangible quantities to achieve a tangible result.

[0063] Unless otherwise specified, and as will be apparent from the description, it is understood that throughout this specification, descriptions utilizing terms such as "processing," "calculating," "computing," "determining," "displaying," or the like may include the actions and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system or other information storage, transmission, or display devices.

[0064] Exemplary embodiments may further relate to apparatuses for performing the operations herein. This apparatus may be specially constructed for the desired purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation containing instructions for performing the operations of a desired embodiment.

[0065] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may eventually be convenient to construct specialized apparatus to perform the desired method steps. Moreover, the example implementations are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the techniques of the example implementations as described herein. Instructions of the programming language may be executed by one or more processing devices, such as, for example, a central processing unit (CPU), a processor, or a controller.

[0066] As known in the art, the operations described above may be performed by hardware, software, or some combination of software and hardware. Various aspects of the exemplary embodiments may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform methods that implement the embodiments of the present application. Furthermore, some exemplary embodiments of the present application may be implemented solely in hardware, while other exemplary embodiments may be implemented solely in software. Furthermore, various functions described may be performed in a single unit or may be distributed across multiple components in any number of ways. When implemented by software, methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted format.

[0067] Additionally, other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings herein. Various aspects and / or components of the described exemplary embodiments may be used singly or in any combination. It is intended that the specification and exemplary embodiments be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.

[0068] Clause 1. A system for automatically identifying subtasks from video data associated with tasks performed by one or more workers and generating labels associated with the identified subtasks and the video data, the system comprising: an edge device including a memory; and at least one processor coupled to the memory and operating at least in part based on information stored in the memory, the at least one processor receiving video data associated with at least one task performed by the at least one worker; executing a human action recognition program to identify at least one action associated with the at least one task based on the received video data; and generating labels associated with the at least one task based on the received video data. and generating at least one label for the identified at least one subtask based on at least one identified action from a first set of existing labels for the identified actions and at least one identified object from a second set of existing labels for the identified objects, wherein the first set of existing labels and the second set of existing labels are stored in a memory.

[0069] Clause 2. The system of clause 1, wherein at least one task is a predefined task associated with an industrial process.

[0070] Clause 3. The system of clause 2, further configured: at least one processor of the edge device calculates a similarity score between the generated at least one label and a task description provided for the predefined task; and determines whether to store the generated at least one label for further analysis or discard it based on the calculated similarity score.

[0071] Clause 4. The system of clause 3, wherein the system is further for analyzing automatically generated labels associated with the identified subtasks to identify an optimal subtask ordering associated with performing the at least one task, the system further comprising: a display; and an analysis device including a second memory and a set of one or more processors coupled to the second memory and based at least in part on information stored in the second memory, the set of one or more processors configured to analyze the set of generated labels for the subtasks of the at least one task for each of a plurality of workers performing the at least one task to determine an optimal subtask ordering associated with performing the at least one task, and present a visual representation of the optimal subtask ordering via the display.

[0072] Clause 5. The system of clause 4, wherein the at least one processor is further configured to generate and store modified video data associated with the at least one task performed by the at least one worker, the modified video data including the received video data and metadata indicating the generated at least one label and a timestamp associated with one or more of a start of the identified at least one subtask, an end of the identified at least one subtask, or a duration of the identified at least one subtask, and wherein the set of one or more processors of the analysis device are configured to analyze the set of generated labels for the subtasks of the at least one task for each of a plurality of workers performing the at least one task by analyzing the modified video data.

[0073] Clause 6. The system of clause 5, wherein the edge device is a first edge device of a plurality of edge devices associated with a corresponding plurality of workstations associated with a plurality of workers performing at least one task, and wherein the at least one processor is configured to provide the modified video data to a common memory associated with the analysis device to store the modified video data.

[0074] Clause 7. The system of clause 1, further comprising a video recording device, the video recording device configured to record video data associated with the at least one task performed by the at least one worker and provide the video data associated with the at least one task performed by the at least one worker to the edge device.

[0075] Clause 8. The system of clause 1, wherein the identified at least one action is associated with at least one of the at least one worker's hand or the at least one worker's wrist.

[0076] Clause 9. The system of clause 8, wherein the identified at least one action includes one or more of reaching for the object, picking up the object, releasing the object, pushing the object, or twisting the object.

[0077] Clause 10. The system of clause 9, wherein, to identify at least one subtask of the at least one task, the at least one processor of the edge device is configured to identify one of a start or an end of the at least one subtask based on a distance of at least one of the at least one worker's hand or the at least one worker's wrist from the at least one identified object.

[0078] Clause 11. An apparatus for subtask identification, comprising: a memory; and at least one processor coupled to the memory and configured to, based at least in part on information stored in the memory, receive video data associated with at least one task performed by at least one worker; execute a human action recognition program to identify at least one action associated with the at least one task based on the received video data; execute an object detection program to identify at least one object associated with the at least one task based on the received video data; identify at least one subtask of the at least one task based on a combination of the identified at least one action associated with the at least one task and the identified at least one object; and generate at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified action and at least a second label associated with the identified at least one object from a second set of existing labels for the identified object, wherein the first set of existing labels and the second set of existing labels are stored in the memory.

[0079] Clause 12. The apparatus of clause 11, wherein at least one task is a predefined task associated with an industrial process.

[0080] Clause 13. The apparatus of Clause 12, further configured: the at least one processor calculates a similarity score between the generated at least one label and a task description provided for the predefined task; and determines whether to store the generated at least one label for further analysis or discard it based on the calculated similarity score.

[0081] Clause 14. The apparatus of clause 13, wherein the apparatus is further configured to analyze automatically generated labels associated with the identified subtasks to identify an optimal subtask ordering associated with performing the at least one task, the apparatus further comprising a display, the at least one processor further configured to analyze the set of generated labels for the subtasks of the at least one task for each of a plurality of workers performing the at least one task to determine an optimal subtask ordering associated with performing the at least one task, and present a visual representation of the optimal subtask ordering via the display.

[0082] Clause 15. The apparatus of clause 14, wherein the at least one processor is further configured to generate and store modified video data associated with the at least one task performed by the at least one worker, the modified video data including the received video data and metadata indicating the generated at least one label and a timestamp associated with one or more of a start of the identified at least one subtask, an end of the identified at least one subtask, or a duration of the identified at least one subtask, and the at least one processor is configured to analyze the set of generated labels for the subtasks of the at least one task for each of a plurality of workers performing the at least one task by analyzing the modified video data.

[0083] Clause 16. The apparatus of clause 15, wherein the apparatus is a first apparatus of a plurality of apparatuses associated with a corresponding plurality of workstations associated with a plurality of workers performing at least one task, and wherein the at least one processor is configured to provide the modified video data to a common memory for storing the modified video data.

[0084] Clause 17. The apparatus of clause 11, wherein the identified at least one action is associated with at least one of a hand of the at least one worker or a wrist of the at least one worker.

[0085] Clause 18. The apparatus of clause 17, wherein the identified at least one action includes one or more of reaching for the object, picking up the object, releasing the object, pushing the object, or twisting the object.

[0086] Clause 19. The apparatus of clause 18, wherein, to identify at least one subtask of the at least one task, the at least one processor is configured to identify one of a start or an end of the at least one subtask based on a distance of at least one of the at least one worker's hand or the at least one worker's wrist from the at least one identified object.

[0087] Clause 20. A method for automatically identifying sub-tasks from video data associated with tasks performed by one or more workers and generating labels associated with the identified sub-tasks and the video data, the method comprising: receiving video data associated with at least one task performed by at least one worker; executing a human action recognition program to identify at least one action associated with the at least one task based on the received video data; executing an object detection program to identify at least one object associated with the at least one task based on the received video data; identifying at least one sub-task of the at least one task based on a combination of the identified at least one action associated with the at least one task and the identified at least one object; and generating at least one label for the identified at least one sub-task based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the identified at least one object from a second set of existing labels for the identified object, wherein the first set of existing labels and the second set of existing labels are stored in at least one memory associated with the human action recognition program and the object detection program.

[0088] 110 Component 111 Screw 120 Product assembly task 310, 330, 350 Edge module 311, 331, 351 Camera 313 First human worker 333 Second human worker 335, 355 Robot arm 400 MES 401, 1010 Work order 410, 420, 430 Edge analysis module 411, 421, 431 Subtask labeling module 440 Labeled image DB 450 Core analysis module 510 Product ID 520 Work cell ID 530 Task number field 531 First task 532 Second task 540 Task ID field 541, 542 Task ID 550 Task description field 551, 552 Task description 610, 810 HAR program 620, 820 Object detection program 630, 930 Subtask identification program 640 Task identification process 650 Task description evaluation module 680 Video labeling module 701 Video data 710 Specific task 715 Specific worker 720 Specific work cell 761, 762 Subtask 771, 772 Subtask ID 781, 782 Subtask description 841 Text DB 842 Label DB 1011 Sentence A 1012, 1052 BERT model 1014 Single vector representation u 1040 Task identification module 1051 Sentence B 1054 Single vector representation v 1210 Video recording device 1220 Edge device 1230 Analysis device 1300 Computing environment 1305 Computer device 1310 Processor 1315 Memory 1320 Internal storage 1325 I / O interface 1330 Bus 1335 Input / user interface 1340 Output device / interface 1345 External storage 1350 Network 1360 Logical unit 1365 API unit 1370 Input unit 1375 Output unit 1395 Inter-unit communication mechanism

Claims

1. A system for automatically identifying subtasks from video data associated with tasks performed by one or more workers and generating identified subtasks and labels associated with the video data, the system comprising: an edge device including a memory; and at least one processor coupled to the memory and based at least in part on information stored in the memory, the at least one processor: receives video data associated with at least one task performed by at least one worker; executes a human action recognition program to identify at least one action associated with the at least one task based on the received video data; executes an object detection program to identify at least one object associated with the at least one task based on the received video data; and identifies at least one subtask of the at least one task based on a combination of the identified at least one action and the identified at least one object associated with the at least one task; generating at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified action and at least a second label associated with the at least one identified object from a second set of existing labels for the identified object, wherein the first set of existing labels and the second set of existing labels are stored in the memory.

2. The system of claim 1, wherein the at least one task is a predefined task associated with an industrial process.

3. The system of claim 2, wherein the at least one processor of the edge device is further configured to: calculate a similarity score between the at least one generated label and a task description provided for the predefined task; and determine whether to store the at least one generated label for further analysis or discard it based on the calculated similarity score.

4. The system of claim 3, wherein the system is further configured to analyze automatically generated labels associated with the identified subtasks to identify an optimal subtask ordering associated with performing the at least one task, the system further comprising: a display; and an analysis device including: a second memory; and a set of one or more processors coupled to the second memory and based at least in part on information stored in the second memory, the set of one or more processors configured to: analyze the set of generated labels for subtasks of the at least one task for each of a plurality of workers performing the at least one task to determine the optimal subtask ordering associated with performing the at least one task; and present via the display a visual representation of the optimal subtask ordering.

5. The system of claim 4, wherein the at least one processor is further configured to generate and store modified video data associated with the at least one task performed by the at least one worker, the modified video data including the received video data and metadata indicating the generated at least one label and a timestamp associated with one or more of a start of the identified at least one subtask, an end of the identified at least one subtask, or a duration of the identified at least one subtask, and the set of one or more processors of the analysis device are configured to analyze the generated set of labels for the subtasks of the at least one task for each of the multiple workers performing the at least one task by analyzing the modified video data.

6. The system of claim 5, wherein the edge device is a first of a plurality of edge devices associated with a corresponding plurality of workstations associated with the plurality of workers performing the at least one task, and wherein the at least one processor is configured to: provide the modified video data to a common memory associated with the analysis device to store the modified video data.

7. The system of claim 1, further comprising: a video recording device configured to: record the video data associated with the at least one task performed by the at least one worker; and provide the video data associated with the at least one task performed by the at least one worker to the edge device.

8. The system of claim 1, wherein the identified at least one action is associated with at least one of a hand of the at least one worker or a wrist of the at least one worker.

9. The system of claim 8, wherein the at least one identified action includes one or more of: reaching for an object, picking up the object, releasing the object, pushing the object, lifting the object, carrying the object, pulling the object, or twisting the object.

10. The system of claim 9, wherein to identify the at least one subtask of the at least one task, the at least one processor of the edge device is configured to identify one of a start or an end of the at least one subtask based on a distance of the at least one of the hand of the at least one worker or the wrist of the at least one worker from at least one identified object.

11. An apparatus for subtask identification, comprising: a memory; and at least one processor coupled to the memory and based at least in part on information stored in the memory, wherein the at least one processor is configured to: receive video data associated with at least one task performed by at least one worker; execute a human action recognition program to identify at least one action associated with the at least one task based on the received video data; execute an object detection program to identify at least one object associated with the at least one task based on the received video data; identify at least one subtask of the at least one task based on a combination of the identified at least one action associated with the at least one task and the identified at least one object; generate at least one label for the identified at least one subtask based on at least a first label associated with the at least one identified action from a first set of existing labels for identified actions and at least a second label associated with the identified at least one object from a second set of existing labels for identified objects, wherein the first set of existing labels and the second set of existing labels are stored in the memory. Device.

12. The apparatus of claim 11, wherein the at least one task is a predefined task associated with an industrial process.

13. The apparatus of claim 12, wherein the at least one processor is further configured to: calculate a similarity score between the at least one generated label and a task description provided for the predefined task; and determine whether to store the at least one generated label for further analysis or discard it based on the calculated similarity score.

14. The apparatus of claim 13, wherein the apparatus is further configured to analyze automatically generated labels associated with the identified subtasks to identify an optimal subtask ordering associated with performing the at least one task, the apparatus further comprising: a display; and the at least one processor is further configured to: analyze a set of generated labels for subtasks of the at least one task for each of a plurality of workers performing the at least one task to determine the optimal subtask ordering associated with performing the at least one task; and present a visual representation of the optimal subtask ordering via the display.

15. The apparatus of claim 14, wherein the at least one processor is further configured to: generate and store modified video data associated with the at least one task performed by the at least one worker, the modified video data including the received video data and metadata indicating the generated at least one label and a timestamp associated with one or more of a start of the identified at least one subtask, an end of the identified at least one subtask, or a duration of the identified at least one subtask, and the at least one processor is configured to analyze the generated set of labels for the subtasks of the at least one task for each of the multiple workers performing the at least one task by analyzing the modified video data.

16. The apparatus of claim 15, wherein the apparatus is a first apparatus of a plurality of apparatuses associated with a corresponding plurality of workstations associated with the plurality of workers performing the at least one task, and wherein the at least one processor is configured to provide the modified video data to a common memory for storing the modified video data.

17. The apparatus of claim 11, wherein the identified at least one action is associated with at least one of a hand of the at least one worker or a wrist of the at least one worker.

18. The apparatus of claim 17, wherein the identified at least one action includes one or more of: reaching for an object, picking up the object, releasing the object, pushing the object, lifting the object, carrying the object, pulling the object, or twisting the object.

19. The apparatus of claim 18, wherein, to identify the at least one subtask of the at least one task, the at least one processor is configured to identify one of a start or an end of the at least one subtask based on a distance of at least one of the hand of the at least one worker or the wrist of the at least one worker from the at least one identified object.

20. A method for automatically identifying sub-tasks from video data associated with tasks performed by one or more workers and generating identified sub-tasks and labels associated with the video data, comprising: receiving video data associated with at least one task performed by at least one worker; executing a human action recognition program to identify at least one action associated with the at least one task based on the received video data; executing an object detection program to identify at least one object associated with the at least one task based on the received video data; identifying at least one sub-task of the at least one task based on a combination of the identified at least one action associated with the at least one task and the identified at least one object; and generating at least one label for the identified at least one sub-task based on at least a first label associated with the at least one identified action from a first set of existing labels for the identified actions and at least a second label associated with the identified at least one object from a second set of existing labels for the identified objects, The first set of pre-existing labels and the second set of pre-existing labels are stored in at least one memory associated with the human action recognition program and the object detection program.

Images