Task management using image capture monitoring of user actions

Abstract

Techniques are disclosed for presenting instructions to a user to complete a task based on monitoring a user action image. The system monitors user actions to identify a next operation of the set of operations for presentation to the user. The system presents instructions to the user to complete the next operation. The system monitors user actions, and may also monitor manufacturing component status or operating equipment status to determine if the operation has been completed. The system may reorder a sequence of operations for a particular task based on one or both of user input and identification of different sequences of operations associated with a higher task execution rating.

Description

[0001] Incorporation by Reference; Disclaimer

[0002] Each of the following applications is hereby incorporated by reference: Application No. 18 / 307,448, filed April 26, 2023; Application No. 63 / 478,326, filed January 3, 2023. Applicant hereby revokes any disclaimer of claim scope in the parent application(s) or its prosecution history and informs the USPTO that the claims in this application may be broader than any claim in the parent application(s). Technical Field

[0003] The present disclosure relates to managing the performance of user tasks by monitoring images of user positions to monitor user actions. Specifically, the present disclosure relates to analyzing user actions recorded with a video monitoring application to determine task characteristics associated with the user actions. Background Art

[0004] In many environments, a user must perform a set of steps to complete a specific task. For example, a user may perform a set of steps to assemble a component in a manufacturing environment. The set of steps may include interacting with equipment in the workspace. The system can provide feedback for completing the set of steps. For example, the system may provide a display indicating what actions the user must perform to complete the operations associated with the specific task. Often, it is difficult for a user to interact with a user interface while completing a set of operations. For example, in a manufacturing environment, the user may be wearing gloves. Furthermore, the sequence of operations may require both hands, so that interrupting the sequence to interact with the user interface will delay the completion of the task.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings. It should be noted that references to "one" or "an" embodiment in this disclosure do not necessarily refer to the same embodiment, and they mean at least one. In the figures:

[0007] Figure 1 illustrates a system according to one or more embodiments;

[0008] Figure 2 illustrates a set of example operations for presenting user task operations based on detected user actions in accordance with one or more embodiments;

[0009] Figure 3illustrates a set of example operations for training a machine learning model to classify recorded user actions into operations associated with a task in accordance with one or more embodiments;

[0010] Figures 4A-4F illustrates example embodiments; and

[0011] Figure 5 A block diagram illustrating a computer system in accordance with one or more embodiments is shown. DETAILED DESCRIPTION

[0012] In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some examples, well-known structures and devices are described in block diagram form to avoid unnecessarily obscuring the present invention.

[0013] 1. General Overview

[0014] 2. System Architecture

[0015] 3. Present user task actions based on detected user actions

[0016] 3.1 Start branch tasks

[0017] 3.2 Triggering the recording of statistics associated with a task

[0018] 3.3 Monitoring Task Indicators Based on Detected User Actions

[0019] 4. Machine Learning Model Training

[0020] 5. Example Embodiments

[0021] 6. Computer Networks and Cloud Networks

[0022] 7. Other; Extension

[0023] 8. Hardware Overview

[0024] 1. General Overview

[0025] One or more embodiments display instructions associated with performing the next operation of a task in response to detecting completion of the current operation of a task via real-time analysis of a video stream of the current operation of (one or more) users completing the task. The system initially displays instructions for performing the current operation of a task in a set of operations. While displaying the instructions for the current operation, the system continuously analyzes the user's video stream in real time while the user performs actions to complete the current operation. Based on the continuous analysis of the video stream, the system detects that the user has completed the current operation. The system uses the detected completion of the current operation of the task as a trigger to switch from displaying instructions for the current operation to displaying instructions for the next operation in the set of operations.

[0026] One or more embodiments generate predictive statistics based on analysis of a video stream of (one or more) users performing a set of task operations. The system analyzes the user's video stream to detect start and / or end times corresponding to the execution of the operations. The start and / or end times of one or more operations are used to generate predictions corresponding to the group of operations. As an example, the system predicts a delay in completing the complete set of task operations based on the detected start time of the current operation of the task, the estimated time to complete the current operation, and the estimated time to complete the operations and tasks to be performed subsequent to the current operation. In another example, the system predicts a delay in completing the complete set of operations based on the detected end time of the current operation and the estimated time to complete the operations to be performed subsequent to the current operation. The system can store the start and / or end times of any statistical information calculations associated with the execution of the operations.

[0027] One or more embodiments perform modifications to a set of tasks to be completed based on real-time analysis of a video stream of a user performing one or more of the tasks. In one example, the system selects a remedial action to be performed in response to determining that the user has performed one or more tasks incorrectly and / or out of sequence. The system can insert and present instructions for performing the remedial action. The system can reorder the remaining tasks based on the user's incorrect and / or out-of-sequence performance of the tasks. The remedial action can undo the performed action, redo the performed action, and / or include compensatory actions that reverse the negative effects of the incorrect and / or out-of-sequence tasks. Alternatively, or in addition, the system can present an alert in response to determining that the user has performed one or more tasks incorrectly and / or out of sequence.

[0028] One or more embodiments record statistical information associated with user performance of tasks based on monitoring user actions. For example, the system may identify a time associated with a user action corresponding to an initial operation of a task. The system may identify another time associated with a user action corresponding to a final operation to complete a task. The system may record the total time required for the user to perform the task. The system may record the statistical information as a metric associated with task performance. For example, the system may maintain a record of how long it takes a user to perform a particular task. The system may maintain metrics that compare a user's performance to that of other users or to the same user's performance over time. One or more embodiments use the recorded statistical information to generate subsequent predictions for operation and task completion times.

[0029] One or more example embodiments use image recognition to identify a specific user in a workspace. The system identifies a set of tasks associated with the user and equipment in the workspace. The system monitors user actions in the workspace in real time via image recording. The system identifies operations associated with the user's actions. The system presents images to the user on a user interface display. These images are associated with additional operations for specific tasks corresponding to the user's actions detected by the system.

[0030] One or more embodiments specify tasks associated with a work sequence in a manufacturing environment. The system specifies operations associated with the steps required to perform the tasks. The system generates instructions for directing a user to perform the operations associated with the steps. The system monitors user actions in real time via image recording to determine initiation and / or completion of operations associated with the tasks.

[0031] One or more embodiments train a machine learning model to identify user actions corresponding to tasks. The system trains the machine learning model using a training dataset of image data corresponding to operator body positioning and sequences of body positioning. The image data may include operator body positioning relative to a workspace. Based on real-time identification of the user's body positioning via image recordings, the system identifies the actions being performed and / or completed by the user.

[0032] One or more embodiments described in this specification and / or recited in the claims may not be included in this general summary.

[0033] 2. System Architecture

[0034] The system 100 includes a workspace monitoring platform 110 that monitors user actions in a workspace 120. The workspace monitoring platform 110 obtains image data from an image capture device 118, such as a video camera. The image capture device 110 captures images of a user 140 in the workspace 120. The workspace 120 is a designated geographic area in which the user 140 performs actions to complete a task. Figure 1 In the example illustrated in FIG, workspace 120 includes resources 123, 124, and 125, as well as operating equipment 121 and 122. Resources 123, 124, and 125 can be used by user 140 in the process of performing a task. For example, a resource can be a portion of component 126 that is added to component 126 during a manufacturing process performed by user 140. Additionally or alternatively, resources 123-125 can be provided to operating equipment 121 and 122 to perform an operation. For example, operating equipment 121 can be a 3D printer, and resource 123 can include a resin used as a material for performing a 3D printing operation.

[0035] The image analysis engine 111 analyzes the image data to identify user actions. According to one embodiment, the image analysis engine 111 identifies user actions by providing the image data to the machine learning model 112. The machine learning engine 115 trains the model 112 to identify user actions in the image data.

[0036] The operation recognition engine 113 identifies the operation associated with the recognized user action. For example, the system can determine that a user pressing a specific button on equipment 121 in the workspace corresponds to an operation of performing a specific operation using the equipment. In addition, the system can determine that a set of user actions, such as selecting resource 123 and attaching resource 123 to component 126, corresponds to a specific operation associated with assembling a manufactured component. According to one embodiment, the operation recognition engine 113 identifies the operation 152 associated with the recognized user action 153 by referring to a mapping of user actions to operations. The mapping can be stored as a table, a directed acyclic graph (DAG) tree, or any other format.

[0037] In some examples, one or more elements of the machine learning engine 115 can use a machine learning algorithm to identify one or both of user actions and task operations based on the image data content. A machine learning algorithm is an algorithm that can be iterated to learn a target model f that best maps a set of input variables to output variables using a set of training data. A machine learning algorithm can include supervised components and / or unsupervised components. Various types of algorithms can be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naive Bayes, k-nearest neighbors, learning vector quantization, support vector machines, bagging, and random forests, boosting, backpropagation, and / or clustering.

[0038] In an embodiment, a set of training data includes a dataset and associated labels. The dataset is associated with the input variables of the target model f (e.g., image data including user positioning data in the workspace). The associated labels are associated with the output variables of the target model f (e.g., specified user actions, specified task operations related to the user actions). The training data can be updated based on, for example, feedback on the accuracy of the current target model f. The updated training data is fed back into the machine learning algorithm, which in turn updates the target model f.

[0039] The machine learning algorithm generates a target model f such that the target model f best fits the dataset of training data to the labels of the training data. Additionally, or alternatively, the machine learning algorithm generates the target model f such that when the target model f is applied to the dataset of training data, a maximum number of outcomes determined by the target model f match the labels of the training data.

[0040] In an embodiment, a machine learning algorithm may be iterated to learn one or both of user actions and operations associated with image data of a user in a workspace. In an embodiment, a set of training data includes image data of a user positioned in a workspace. The image data is associated with labels indicating one or both of user actions and user operations.

[0041] In an example, the system initially trains a neural network using a historical dataset. Training the neural network includes generating n hidden layers for the neural network and functions / weights applied to each hidden layer to calculate the next hidden layer. Training may further include determining the function / weight of the final nth hidden layer to be applied to calculate the final label(s) or prediction(s) of the data point.

[0042] Training a neural network includes: (a) obtaining a training data set; (b) iteratively applying the training data set to the neural network to generate labels for data points in the training data set; and (c) adjusting weights and biases associated with the formulas of the neurons that make up the neural network based on a loss function that compares the values ​​associated with the generated labels with the values ​​associated with the test labels. The neurons of the neural network include an activation function to specify the boundaries of the values ​​output by the neurons. The activation function can include a differentiable nonlinear activation function, such as a rectified linear activation (ReLU) function, a logistic function, or a hyperbolic tangent function. Each neuron receives the value of each neuron in the previous layer, applies a weight to each value of the previous layer, and applies one or more biases to the combined value of the previous layer. The activation function limits the range of possible output values ​​from the neuron. The sigmoid activation function converts the neuron value to a value between 0 and 1. The ReLU activation function converts the neuron value to 0 if the neuron value is negative, and to the output value if the neuron value is positive. The ReLU activation function can also be scaled to output a value between 0 and 1. For example, after applying weights and bias values ​​to the values ​​from the previous layer of one neuron, the system can scale the neuron value to a value between -1 and +1. The system can then apply a ReLU-type activation function to generate a neuron output value between 0 and 1. The system trains the neural network using the training, testing, and validation datasets until the labels generated by the trained neural network are within a specified level of accuracy, such as 98% accuracy.

[0043] The task monitoring engine 114 identifies tasks associated with the identified operation. Task 151 is a group of one or more operations. A task can specify operations that should be performed in a specific order. A task can also specify operations that are not required to be performed in a specific order. The task monitoring engine 114 can identify tasks based on detecting a specific operation or a specific group of operations. According to one embodiment, the task monitoring engine 114 identifies tasks associated with the identified operation or group of operations by referring to a mapping of tasks 151 to operations 152. The mapping can be stored as a table, a directed acyclic graph (DAG) tree, or any other format.

[0044] The task monitoring engine 114 determines whether the user action identified by the image analysis engine 111 corresponds to a specific operation of the task. For example, if the task includes operations A, B, and C, and if the task monitoring engine 114 determines that the user performed operation A, the task monitoring engine 114 monitors the next user action to determine whether the next user action corresponds to operation B or operation C.

[0045] When user 140 is performing a specific task, task monitoring engine 114 displays subsequent operations to be performed by user 140 on user interface 117. For example, if task monitoring engine 114 determines that the user has completed operation B in a set of operations, task monitoring engine 114 may cause user interface 117 to display an image corresponding to the next operation in the set of operations. In addition, if task monitoring engine 114 determines that the detected user action corresponds to a system operation trigger, task monitoring engine 114 initiates a system operation. For example, task monitoring engine 114 may determine that the detected user action does not correspond to an operation in the currently executing task. Task monitoring engine 114 may perform operations such as generating an error notification, generating a prompt to add a new operation corresponding to the detected user action to the task, or initiating a branch task to correct the user error.

[0046] User 140 interacts with user interface 117 to perform tasks. For example, user 140 can interact with user interface 117 to select a task to perform. Additionally or alternatively, user 140 can interact with user interface 117 to confirm changes to a particular task. User interface 117 includes a display device that presents instructions to user 140 for completing an operation. These instructions may include textual content and / or graphical content. According to one embodiment, task monitoring engine 114 controls user interface 117 to present instructions to user 140 for performing the next operation in a set of operations.

[0047] The task metric recording engine 116 records metrics 154 associated with tasks. Examples of task metrics include completion time, error detection rate, other anomaly detection rate, equipment utilization rate, and equipment failure detection rate. For example, the system may detect the start time and end time of a task or operation. The system may store the task or operation completion time. The system may compare the completion time with the expected completion time for the same user or other users. Based on this comparison, the system may provide efficiency metrics for a specific user, specific operation, or specific task to the user or other entity.

[0048] The workspace metrics application platform 130 processes the metrics generated by the task metrics recording engine 116 to perform additional functions associated with the workspace 120. For example, the workspace metrics application platform 130 may schedule the operating equipment 121 for maintenance based on detecting a particular usage level or a particular error rate level associated with operations performed using the operating equipment 121. The workspace metrics application platform 130 may generate an operator ranking based on completion time, error rate, or productivity metrics associated with one or more users.

[0049] In one or more embodiments, system 100 may include Figure 1More or fewer components may be used than those shown in the figure. Figure 1 The components illustrated in the may be local or remote from each other. Figure 1 The components illustrated in the figure can be implemented in software and / or hardware. Each component can be distributed across multiple applications and / or machines. Multiple components can be combined into one application and / or machine. Operations described with respect to one component can be performed by another component instead.

[0050] Additional embodiments and / or examples related to computer networks are described below in Section 4 entitled "Computer Networks and Cloud Networks."

[0051] In one or more embodiments, the data repository 150 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). Further, the data repository 150 may include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not be of the same type or located at the same physical site. Furthermore, the data repository 150 may be implemented or executed on the same computing system as the workspace monitoring platform 110. Alternatively, or in addition, the data repository 150 may be implemented or executed on a computing system separate from the workspace monitoring platform 110. The data repository 150 may be communicatively coupled to the workspace monitoring platform 110 via a direct connection or via a network.

[0052] The information describing the mapping of actions to operations and operations to tasks may be implemented across any of the components within system 100. However, for purposes of clarity and explanation, this information is illustrated within data repository 150.

[0053] In one or more embodiments, the workspace monitoring platform 110 refers to hardware and / or software configured to perform the operations described herein for capturing images of user actions in a workspace and performing actions (such as presenting instructions for performing the next action) based on the detected user actions. Figure 2 An example of an operation for presenting instructions for performing a series of operations based on detecting user actions in image data is described.

[0054] In an embodiment, the workspace monitoring platform 110 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, special-purpose hardware devices, hardware routers, hardware switches, hardware firewalls, hardware firewalls, hardware network address translators (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handheld terminals, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management equipment, routers, switches, controllers, access points, and / or client devices.

[0055] In one or more embodiments, interface 117 refers to hardware and / or software configured to facilitate communication between a user and workspace monitoring platform 110. Interface 117 presents user interface elements and receives input via the user interface elements. Examples of interfaces include graphical user interfaces (GUIs), command line interfaces (CLIs), tactile interfaces, and voice command interfaces. Examples of user interface elements include check boxes, radio buttons, drop-down lists, list boxes, buttons, toggle keys, text fields, date and time selectors, command lines, sliders, pages, and forms.

[0056] In an embodiment, different components of interface 117 are specified in different languages. The behavior of user interface elements is specified in a dynamic programming language (such as JavaScript). The content of user interface elements is specified in a markup language such as Hypertext Markup Language (HTML) or XML User Interface Language (XUL). The layout of user interface elements is specified in a style sheet language (such as Cascading Style Sheets (CSS)). Alternatively, interface 117 is specified in one or more other languages ​​(such as Java, C or C++).

[0057] 3. Present user task actions based on detected user actions

[0058] Figure 2 Illustrated are a set of example operations for presenting user task operations based on detecting user actions in accordance with one or more embodiments. Figure 2 One or more of the operations illustrated in the figure may be modified, rearranged or omitted altogether. Figure 2 The particular sequence of operations illustrated in the specification should not be construed as limiting the scope of one or more embodiments.

[0059] The system identifies a task consisting of a set of operations (operation 202). For example, a user may select a user interface icon associated with a task. Based on the user selection, the system identifies a set of operations to be performed by the user to complete the task. Additionally, the set of operations may further include one or more operations to be performed by the machine or by another user. Alternatively, or in addition, the system may analyze image data of the user's actions (such as data recorded by a video recorder) to identify the actions or groups of actions performed by the user. For example, a user may interact with a machine by selecting a component from a box. The system may identify the action as belonging to a task of "assembling a device module." The task may include six operations, including the operation of selecting a component from a box. According to an example embodiment, a workspace in a manufacturing facility includes one or more pieces of equipment. The system identifies the task consisting of the set of operations based on detecting user actions in the workspace (such as a user interacting with a particular piece of equipment) in recorded video data.

[0060] According to an example embodiment, a system identifies tasks based on detecting patterns of user actions. For example, one task may include cleaning subassemblies. Another task may include assembling subassemblies into an assembly. The system may identify which task a user is initiating based on detecting different patterns of user actions. Detecting cleaning a subassembly may include detecting a user placing an item on a surface and reaching for a cleaner. Detecting assembling a subassembly into an assembly may include detecting a user placing an item on a surface and reaching for another subassembly that constitutes the assembly. The system may learn patterns associated with one or more actions corresponding to specific tasks.

[0061] The system presents an operation from a set of operations that constitute a task via a user interface (operation 204). For example, in a set of ten operations, the system identifies the next operation to be performed in the set of ten operations. The set of operations can be arranged in a predefined sequence. Alternatively, the task can be completed by performing operations in multiple different sequences or in any sequence. One or more operations may be optional for completing the task. Completing the task may require other operations. In examples where a set of operations is required to be performed in a specific sequence, the system identifies the next operation in the sequence and presents the next operation in the sequence. In examples where multiple different operations can be performed next (e.g., either as the first operation in the task or as an operation subsequent to an operation corresponding to a recognized user action), the system selects the operation to present based on predefined criteria. For example, the system may determine that performing operations in one sequence may result in a better execution time than performing operations in another sequence. Based on the sequence associated with the better execution time, the system may present the operations via the user interface. The system may determine that resources required for one operation are available, while resources required for another operation are not available. Similarly, the system may predict that performing operations in one sequence will result in delays due to waiting for resources (such as machines or subcomponents) to become available. If there are no explicit time-based or resource-based criteria for selecting an operation to present, the system may select an operation randomly or semi-randomly. For example, the system may select an operation to present based on the memory address of a file associated with the display data for the operation, a number randomly assigned to the operation, or an alphabetical order associated with the operation title.

[0062] The presenting operation may include displaying a photograph or graphic image of the equipment or component to be handled by the user, as well as the action to be performed by the user. For example, the system may display a photograph of the test equipment, a subassembly placed at a specific location on the test equipment, and a highlighted icon of a button to be pressed to initiate a test of the subassembly. According to an example embodiment, the presenting operation includes presenting a real-time video image of the component or equipment. As the user interacts with the component or equipment, the system may show a video image of the user's hands and the component or equipment, as well as a graphic image of the action to be performed using the component or equipment.

[0063] The system detects the execution of a user action (operation 206). According to an exemplary embodiment, the system detects the execution of the user action based on image data of the user in the workspace. The image data can be acquired from a single camera or multiple cameras. The system can detect the execution of the user action based on the image data in combination with additional data, such as sensor data that detects a weight change in a box containing subassemblies, powering on the equipment, or a temperature change in the equipment.

[0064] In one embodiment, the system presents a video image of the equipment or subassembly associated with the operation to be performed, along with the operation to be performed. For example, the operation may include placing the subassembly within a container. The system may provide a video image of the container along with an icon indicating the subassembly has been placed within the container. The user may observe the user performing the presented operation via the video image of the container in the workspace. As the user interacts with the container corresponding to the operation, the user may see the interaction in the displayed video image.

[0065] The system can detect user actions via an action-recognition application. According to one or more embodiments, the system trains a machine learning model to recognize user actions. The system can train the machine learning model using a dataset of images depicting user positioning within a workspace. These images can include the user interacting with specific components (such as manufactured components) and with specific equipment. The system can train the machine learning model to recognize different actions associated with the same component and the same equipment. As an example, a set of images depicting one user action can show the user pressing a button on the equipment to place the equipment in a desired setting. Another set of images depicting another user action can show the user moving the component to the equipment. Yet another set of images depicting another user action can show the user interacting with a specific actuation mechanism to cause the equipment to modify the component for a specific duration, such as by welding a subassembly to the component. The system can train the machine learning model to recognize different sets of images showing separate user actions.

[0066] The system matches the identified user actions with the operations of one or more tasks. The machine learning model can be trained to match actions with task operations. Alternatively, the machine learning model can be trained to recognize actions, and the mapping engine can map actions and action groups to operations. For example, an operation (setting a piece of equipment to a specific setting) can be part of three stored tasks. Each task may be associated with a different manufacturing component. The system can identify a set of user actions as corresponding to a set of operations for a specific task by identifying the following operations: (a) a specific operation associated with multiple tasks, and (b) another operation corresponding to a detected user action of processing a specific manufacturing component. The system can identify the combination of operations (a) and (b) as being associated with a specific task. If the user instead selects another manufacturing component (e.g., operation (c)), the system can identify the combination of operations (a) and (c) as being associated with a different specific task.

[0067] The system determines whether the detected user action corresponds to a system operation trigger (operation 208). In an example embodiment where a task consists of a set of operations to be performed in a specified sequence, the system can determine that (a) the user selected the task for execution or started the execution of the task, and (b) the user performed an operation within the set of operations constituting the task, but the execution was not in sequence. According to another example, the system can determine that the user performed the operation incorrectly. The system can detect a change in the user's action that corresponds to the specified action of the operation. Alternatively, the system can detect an abnormal state of an equipment or manufactured component.

[0068] According to yet another example, the system may determine that the user performed an action that is not included in the set of actions corresponding to the task. According to yet another example, the system may determine that the user completed the last action for a particular task. According to yet another example, the system may determine that multiple possible actions can be performed subsequent to a previously performed action.

[0069] If the system determines that the user action does not correspond to a system action trigger, the system identifies one or more subsequent actions for the particular task (action 210). For example, if the observed user action corresponds to an action that is one of a sequence of actions in the task, the system identifies the next action in the sequence of actions. According to another example, the observed user action may correspond to a task action that is one of a set of actions that are not associated with any particular sequence. For example, a task may include actions A, B, and C that can be completed in any order. When the actions are not associated with any particular sequence, the system may select an action as a subsequent action based on one or more execution criteria.

[0070] For example, the system can calculate which sequence of operations meets a specific execution threshold, such as: the probability of successful execution of a task, or the shortest time to complete a task. According to another example, the system can select an operation as a subsequent operation based on analyzing past executions of a task to identify which operation is most often selected as a subsequent operation by the user or other users. According to yet another example, the system can select an operation as a subsequent operation based on analyzing one or both of the equipment status and the manufacturing component status. For example, if a particular equipment is being used by another user, the system can avoid selecting an operation that requires the equipment as a subsequent operation. If the equipment requires warm-up time, the system can select an operation to start warming up the equipment so that another operation can be performed while the equipment is warming up. According to another example, if a particular manufacturing component is temporarily in short supply or out of stock, the system can avoid selecting a particular operation that requires the manufacturing component as a subsequent operation.

[0071] The system presents subsequent operations to the user via the user interface (operation 212). According to one example, the system presents the subsequent operations to the user without any intervening instructions from the user. For example, the user does not need to press a button or icon to advance the display of the user interface from one operation to the next. Instead, the system automatically advances the display from one operation to the subsequent operation based on observing the user actions via an image capture device and identifying the operations and tasks associated with the user actions. According to one or more embodiments, the user actions are not user actions that interact with the user interface. Instead, these actions are actions that interact with equipment and / or components in the workspace.

[0072] According to one embodiment, the system presents subsequent actions via the user interface by displaying instructions for completing the next action. For example, the system may display icons via the user interface that represent a user physically performing a specific action associated with a specific piece of equipment and / or a specific manufacturing component. Additionally or alternatively, the system may display textual instructions for performing the user action corresponding to the subsequent action. Additionally or alternatively, the system may present instructions for performing the user action corresponding to the subsequent action via voice or audio transmission.

[0073] According to one or more embodiments, the presentation of subsequent operations may include a description of the component to be manipulated by the user during the operation. The description may show how the component is to be modified. The description may show the final state of the component after being modified.

[0074] According to one or more embodiments, the system presents subsequent actions based on determining that the user has completed a previous action. For example, the system may observe that an action has been completed by the user's body positioning. Additionally, or alternatively, the system may obtain status data from the equipment that indicates that the action performed by the user has caused the equipment to acquire a specific state (such as, powered on, testing completed, manufacturing operations completed, etc.). Alternatively, the system may identify the state of the component operated on by the user. The system may scan the component to determine that the user has performed a modification. For example, the system may identify via video analysis that component A has been attached to component B. As another example, the system may detect a weight change on a work surface that indicates the presence of component A on the work surface.

[0075] According to alternative embodiments, the system may present subsequent operations based on determining that a previous operation was initiated by a user. According to one or more embodiments, the system may present subsequent operations based on detecting a particular state of a manufactured component, a resource used to manufacture the component, or a state of equipment in a workspace.

[0076] If the system determines that the user action identified in operation 206 corresponds to a system operation trigger, the system performs the operation associated with the trigger (operation 214). For example, if the system detects that the user action identified in operation 206 is the last operation of a task, the system can perform one or more operations associated with the completion of the task, such as: recording the time it took the user to complete the task, recording any errors or exceptions detected during the user's task execution, recording the completion of the task in the task management system (which may trigger an option in another user's interface to perform a dependent task), and identifying the next task to be performed by the user. As an example, the system can identify a task mode that is typically performed by the user. After one task in the mode is completed, the system can display an operation on the user interface to start another task of the task mode.

[0077] Another system operation trigger includes determining that the user performed operations out of sequence. In response, the system may (a) recommend terminating the task, (b) recommend restarting the task, (c) generate a notification on the user interface that the operations were detected out of sequence, or (d) prompt the user to indicate whether to generate a new sequence of operations for the task based on the user's current sequence of performed operations. The system may select from the above system operations (a)-(d) based on one or both of previously generated rules or detected task characteristics. For example, if the system determines that the task includes sequential operations A, B, C, and D, and the system detects that the user performed operation C before operation B, the user may further determine that such a sequence change will result in the manufactured component being manufactured incorrectly. Therefore, the system may determine that the sequence cannot be rearranged and that the system operation must include a prompt to suspend and restart the task. Alternatively, if the system determines that the task includes sequential operations A, B, C, and D, but performing the operations out of sequence will not result in a known defect in the manufactured component, the system may prompt the user to indicate whether to store a new sequence for performing the operations.

[0078] Another system operation trigger includes detecting that a performed user action does not match a user action associated with a particular operation (such as a currently displayed operation). For example, the system may detect via an imaging sensor that a user has moved an actuator of the equipment by 45 degrees, but the operation specifies moving the actuator by 90 degrees. The system may (a) recommend aborting the task; (b) recommend repeating the operation; (c) recommend restarting the task; (d) generate a notification on the user interface that the detected user action does not correspond to the displayed user action; or (d) prompt the user to indicate whether to modify the action set of the stored operation to correspond to the currently detected user action. The system may select from the above system operations (a)-(d) based on one or both of previously generated rules or characteristics of the detected task and / or operation.

[0079] Another system operation trigger includes detecting the state of equipment or resources associated with a task. For example, the system may detect that the equipment is in an error state. Alternatively, the system may detect that the equipment is not in an error state, but is also not in a state specified for a particular task. The system may perform a system operation that recommends suspending or restarting the operation or task, or perform one or more intervention operations to place the equipment in a state specified for the task before continuing the operation specified in the task. Additionally, or alternatively, the system may detect an abnormal state of a manufactured component associated with the task and / or operation. The operation may include a user action of placing the component in an analytical device (such as a scanner). Based on the detection of an abnormality in the component, the system may recommend suspending or restarting the task or operation. Alternatively, the system may display one or more recommended operations for correcting the detected abnormality. For example, if the abnormality includes a pipe being fastened to a flange, the system may recommend an operation to tighten the fastening device before continuing the operation for manufacturing the component.

[0080] Another system operation trigger includes determining that multiple possible operations can be performed subsequent to a previously performed operation. For example, three different tasks may begin with operations A and B. The system may prompt the user to select which of the three tasks the user is currently performing to allow the system to recommend an appropriate subsequent operation for that task. For example, the prompt may be presented via a touch interface and / or an audio interface.

[0081] According to one or more embodiments, performing an action associated with a trigger includes changing a state of an equipment without user intervention in response to detecting a user action. For example, the system may detect a user action of obtaining component A from box A. The system may illuminate a light on box B corresponding to the next action to be performed (e.g., selecting component B from box B) without user intervention. Alternatively, the system may determine that selecting component A is an action that is not performed in sequence. Therefore, the system may illuminate a light on box A (such as a red light) and another light on box B (such as a green light) to prompt the user to correct the sequence of operations. The system may initiate a warm-up sequence for the equipment associated with the next action in the sequence of actions. According to another example, in response to detecting a user action of placing a component in the equipment, the system may cause the equipment to initiate a scan for the component.

[0082] After executing the system operation associated with the system operation trigger, the system determines whether a subsequent operation exists (operation 216). For example, the system operation may include generating a notification to the user or generating a prompt for user input. After receiving the notification or responding to the prompt, the user may still continue to perform the operation associated with the task. Therefore, the workflow continues to operation 208. Conversely, the system operation may include terminating the task. If the system determines that the detected operation is the last operation in the task, the system may avoid displaying information about any subsequent operations. For example, the system may generate a user interface element indicating that the task has been completed. If the system determines that the detected fault or anomaly makes the task impossible to perform or the execution is unsatisfactory, the system may avoid displaying additional operation information. The system may generate a user interface element prompting the user to restart the task, start a new task, or obtain help from another operator. The additional operator may be in a different workspace than the operator being monitored by the workspace monitoring platform 110. For example, the additional operator may be in an adjacent workspace or may be a supervisor of the monitored operator. According to one example, an embodiment monitors multiple different operators in different workspaces or the same workspace. For example, two operators may be operating in the same workspace to assemble a product. The system can monitor the operations of each operator independently. As another example, the system can monitor the actions of operators in two separate workspaces.The system can generate notifications to additional operators based on the operator's status (such as whether the operator is taking a break between tasks or operations).

[0083] 3.1 Start branch tasks

[0084] According to one or more embodiments, the system initiates a branch task that branches off from a main task being performed by a user. For example, the system may detect a user action that differs from a displayed action for a particular operation. Based on the detected difference between the user action and the displayed user action, the system may initiate a branch task. For example, a main task may include operations A, B, and C. The system may detect that the user performed operation D instead of operation B. The system may initiate a branch task that prompts the user to perform operations D1 and D2. The system may then return to the main task by prompting the user to perform operation C.

[0085] As an example, operation D may include the user pressing a button on the machine instead of another button designated for the task. Operations D1 and D2 may correspond to user actions interacting with the machine to prepare it for operation C. According to an alternative example, operations D1 and D2 may include an instructional video illustrating how to perform operation B. The system may then prompt the user to repeat operation B or restart the task.

[0086] According to another example, a user can start a task that includes operations A, B, C, and D to be performed in sequence. After performing operation B, the system displays a graphical representation associated with operation C on the user interface. A supervisor can assign an urgent task to the user. Therefore, the system detects that the user performs operation E associated with the urgent task. The system can determine (a) whether the first operation must be suspended, or (b) whether the user's progress is stored in the first task for completion at a later time. For example, the system can determine that operation E is part of a sequence of operations E, F, B, and G. If operation B requires the same equipment in two tasks, the system can suspend the first task to allow the user to access the equipment in the second task. Alternatively, the system can determine that the new task does not conflict with the initial task. Therefore, the system can display the operation of the new task as a branch task from the initial task. Then, after detecting that operation G is completed by the user, the system can resume prompting the user to perform the initial task by returning to display the graphic associated with operation C of the initial task.

[0087] 3.2 Triggering the recording of statistics associated with a task

[0088] According to one or more embodiments, performing operations and tasks causes the system to record statistics associated with the operations and tasks. These statistics may, for example, include recording the resources consumed to perform the operation or task. These statistics may further include, for example, recording the number of components manufactured as a result of the task completion. According to another example, the system records usage statistics for the machinery used to perform the operations and tasks. For example, the system may track how long the machinery is operated to perform the operations and tasks. This usage information may be fed into a maintenance log for scheduling equipment maintenance.

[0089] According to one or more embodiments, the system uses task statistics to generate and / or modify a schedule for the execution of future tasks and / or operations. For example, the system can use task metrics associated with one or more tasks to: calculate future delays and their production lines, plan to ramp up resources at specific times in the future, predict the completion time of future tasks, and predict the completion time of projects that depend on one or more tasks.

[0090] 3.3 Monitoring Task Indicators Based on Detected User Actions

[0091] According to one or more embodiments, the system generates task metrics based on detected user actions. Examples of task metrics include: completion time, error detection rate, other anomaly detection rate, equipment utilization rate, and equipment failure detection rate. For example, the system can detect the start time of a task or operation and the end time of the task or operation. The system can store the task or operation completion time. The system can compare the completion time with the expected completion time based on the same user or other users. Based on the comparison, the system can provide efficiency metrics for a specific user, specific operation, or specific task to the user or other entity. For example, the system can generate a task completion rating for a group of operators performing the same task.

[0092] 4. Machine Learning Model Training

[0093] Figure 3 Illustrated is a set of example operations for training a machine learning model to classify patterns of user actions into operations and / or tasks corresponding to one or more embodiments. Classifying user actions can include classifying them according to (a) a classification corresponding to a specific task, and / or (b) a classification corresponding to a specific operation. According to one embodiment, the system also assigns confidence levels to predictions. Based on one user action, the system can assign a low confidence level to a specific classification corresponding to a specific task. Based on two or more user actions performed in sequence, the system can assign a higher confidence level to a specific classification for a specific user action.

[0094] The method includes identifying or acquiring historical video image data (operation 302). Acquiring historical data may include acquiring data associated with a user's location within a workspace. For example, a set of image data may include a user's interaction with equipment, components, or another user within the workspace. The historical data may be associated with other data, such as equipment status data (e.g., whether a particular machine was turned on or in a particular state when a particular set of image data was captured). The historical image data may include real image data (e.g., captured by a video camera) and synthetic image data (e.g., data generated by a computer to simulate video data).

[0095] The system uses historical video data to generate a training data set (operation 304). The training data set includes video data of recorded user actions and labels identifying operations and / or tasks associated with the recorded user actions. The training data set may include additional attributes associated with the video data, including: the identity of the user, the location of the user within the organization, previous operations and / or tasks performed by the user before the recorded action, the location of the workspace in which the user is operating, equipment data (such as the status of machines in the workspace (e.g., powered on, powered off, ready for operation, cooling down from operation, performing operation)). The training labels may include: identifiers of operations (e.g., sub-processes of a task consisting of multiple operations) associated with the action (e.g., "operation: connect component A to component B"), and / or identifiers of tasks associated with the action and / or corresponding operation (e.g., "operation: connect component A to component B; task: assemble component XYZ").

[0096] In some embodiments, generating a training dataset includes generating a set of feature vectors for labeled examples. The feature vectors for the examples can be n-dimensional, where n represents the number of features in the vector. The number of features selected can vary depending on the specific implementation. Features can be curated in a supervised approach or automatically selected from extracted attributes during model training and / or tuning. Example features include the user's identity, the user's location within the organization, previous actions and / or tasks performed by the user before the recorded action, the workspace location in which the user is operating, and equipment data (such as the status of equipment in the workspace). In some embodiments, features within the feature vectors are represented in numerical form by one or more bits. The system can use encoding schemes such as one-hot encoding, label encoding, and binary encoding to convert categorical attributes into numerical representations. One-hot encoding creates a unique binary feature for each possible category in the original features. In one-hot encoding, when one feature has a value of 1, the remaining features have a value of 0. For example, if a type of healthcare service has ten different categories, the system can generate ten different features for the input dataset. When one category (e.g., a value of "1") is present, the remaining features are assigned a value of "0." According to another example, the system can perform label encoding by assigning a unique numerical value to each category.According to yet another example, the system can perform binary encoding by converting the numerical value into binary digits and creating a new feature for each digit.

[0097] The system applies a machine learning algorithm to the training data set to train a machine learning model (operation 306). The machine learning algorithm analyzes the training data set to train neurons of a neural network with specific weights and biases to associate specific recorded user actions with specific operations and / or tasks. According to one or more embodiments, the machine learning algorithm or post-machine learning algorithm further assigns a confidence score to the prediction. For example, training the machine learning model can identify a relationship between a user action and a specific operation that is part of three different tasks. Training the machine learning model can further determine that when the operation is performed after task A is completed, the operation is most likely associated with an operation in task B. Training the machine learning model can further determine that when the operation is performed after task C is completed, the operation is most likely associated with an operation in task D. Therefore, if the machine learning model determines that the operation identified in the recorded user action is performed after task C, then the operation is likely to be operation P in task D with an 80% confidence level.

[0098] In some embodiments, the system iteratively applies a machine learning algorithm to a set of input data to generate a set of output labels, compares the generated labels to pre-generated labels associated with the input data, adjusts the algorithm's weights and biases based on errors, and applies the algorithm to another set of input data. In some cases, the system can generate and train a candidate recurrent neural network model, such as a long short-term memory (LSTM) model. For recurrent neural networks, one or more network nodes, or "cells," can include memory. Memory allows individual nodes in the neural network to capture dependencies based on the order in which feature vectors are fed through the model. The weights applied to a feature vector representing an expense or activity can depend on the feature vector's position within the sequence represented by the feature vectors. Thus, a node can have a memory to remember the temporal dependencies between different recorded user actions. For example, a recorded user action can have a first set of weights applied by a node as a function of the corresponding feature vector for the expense. However, if a recorded user action is immediately preceded by another type of recorded user action associated with an action in a particular task, a different set of weights can be applied by one or more nodes based on the memory of the preceding recorded user action. In this case, the action prediction assigned to the second recorded user action can be influenced by the first recorded user action. Additionally, or alternatively, the system can generate and train other candidate models, such as support vector machines, decision trees, Bayesian classifiers, and / or fuzzy logic models, as previously described.

[0099] In some embodiments, the system compares the labels estimated by one or more iterations of the machine learning model algorithm with the observed labels to determine an estimation error (operation 308). The system can perform this comparison on a set of test examples, which can be a subset of examples in the training dataset that were not used to generate and fit the candidate model. The total estimation error for a particular iteration of the machine learning algorithm can be calculated as a function of the size of the difference and / or the number of examples for which the estimated label was incorrectly predicted.

[0100] In some embodiments, the system determines whether to adjust weights and / or other model parameters based on the estimated error (operation 310). Adjustments can be made until a candidate model is identified that minimizes the estimated error or otherwise reaches a threshold level of estimated error. The process can return to operation 308 to make adjustments and continue training the machine learning model.

[0101] In some embodiments, the system selects machine learning model parameters based on the estimated error meeting a threshold accuracy level (operation 312). For example, the system can select a set of parameter values ​​for the machine learning model based on determining that the trained model has an accuracy level of at least 98% for predicting labels for actions and / or tasks for recorded user actions.

[0102] In some embodiments, the system uses backpropagation to train neural networks. Backpropagation is the process of updating the states of units in a neural network based on the gradient of a function of the estimated error. Through backpropagation, nodes are assigned a fraction of the estimated error based on their contribution to the output and are adjusted based on this fraction. In recurrent neural networks, time is also factored into the backpropagation process. As previously mentioned, a given example may include a sequence of related recorded user actions. Each recorded user action can be processed as a separate discrete time instance. For example, an example may include recorded user actions c1, c2, and c3 corresponding to times t, t+1, and t+2, respectively. Backpropagation through time can perform adjustments by gradient descent starting at time t+2 and moving backward in time to t+1 and then to t. Furthermore, the backpropagation process can adjust the memory parameters of units so that the units remember the contributions from previously recorded user actions in the sequence of recorded user actions. For example, a unit calculating the contribution of e3 can have a memory of the contribution of e2, which in turn has a memory of e1. Memories can be used as feedback connections so that the output of a unit at one time (e.g., t) is used as input at the next time in the sequence (e.g., t+1). Gradient descent techniques can take these feedback connections into account so that the contribution of one recorded user action to the output of a unit can influence the contribution of the next recorded user action to the output of the unit. Thus, the contribution of c1 can influence the contribution of c2, and so on.

[0103] Additionally or alternatively, the system can train other types of machine learning models. For example, the system can adjust the boundaries of a hyperplane in a support vector machine or the weights of nodes in a decision tree model to minimize estimation error. After training, the machine learning model can be used to estimate labels for new examples of recorded user actions.

[0104] In an embodiment where the machine learning algorithm is a supervised machine learning algorithm, the system may optionally obtain feedback on various aspects of the above analysis (operation 314). For example, the feedback may assert or revise the labels generated by the machine learning model. The machine learning model may indicate that a particular recorded user action is associated with a label of "connecting component A to component B". The system may receive feedback indicating that a particular recorded user action should instead be associated with a label of "disconnecting component A from component B". Based on the feedback, the machine learning training set may be updated to improve its analysis accuracy (operation 316). After being updated, the system may further train the machine learning model by optionally applying the model to additional training data sets.

[0105] 5. Example Embodiments

[0106] For the purpose of clarity, detailed examples are described below. The components and / or operations described below should be understood as a specific example that may not be applicable to certain embodiments. Therefore, the components and / or operations described below should not be interpreted as limiting the scope of any one of the claims.

[0107] Figure 4A A user interface display 470 is shown showing a set of tasks 471 that can be performed at a workstation. For example, a workstation management system can detect the presence of a particular user and display a set of tasks that the particular user can perform at the workstation. Based on the user's selection of the task "Assemble Widget," the task is performed by Figure 4A The box around the task name in represents that the system displays a set of operations 472 to be performed to complete the task, such as Figure 4B As shown in the figure.

[0108] Figure 4B The set of operations 472 and the currently displayed operation 473, "Get component A from bin 14," are illustrated. The system further displays a depiction of bin 14 (reference numeral 474) and a depiction of component A (reference numeral 475).

[0109] refer to Figure 4C, the system detects user actions via video camera 428. Workspace 420 is monitored by camera 428. Camera 428 monitors the actions of user 440 in workspace 420. Workspace 420 also includes equipment 421 and 422, work surface 426, and display device 470. Workspace monitoring platform 410 acquires camera data and state data from equipment 421 and 422 to detect user actions. For example, the system detects user actions of moving to box 424 and from box 424 to work surface 426. The system also detects changes in the weight of box 424. The system provides video and sensor data to user action classification machine learning model 411. Model 411 determines that the user action corresponds to the first operation of a task (e.g., "Step 1: Get component A from box 14") (operation 412). In response to identifying the user action as corresponding to the first operation in the task, the system displays the next operation in the task (e.g., "Operation 2") on display device 470 (operation 413).

[0110] refer to Figure 4D , the system detects the next user action via the video camera 428. The workspace monitoring platform 410 provides the video and sensor data to the user action classification machine learning model 411. Model 411 determines that the user action corresponds to operation 3 of the task, rather than the displayed action associated with operation 2 of the task (operation 414). In response to identifying the user action as corresponding to the third operation in the task, the system determines whether the operations that constitute the task can be performed out of sequence. Based on determining that the operations of the task can be performed out of sequence (operation 415), the system identifies the next operation to be displayed. The system determines that the second operation (operation 2) should be performed next and redisplays the user action associated with the second operation on the display device 470 (operation 416).

[0111] refer to Figure 4E, the system detects the next user action via video camera 428. The workspace monitoring platform 410 provides the video and sensor data to the user action classification machine learning model 411. Model 411 determines that the user action corresponds to operation 4 of the task, rather than the displayed action associated with operation 2 of the task (operation 417). In response to identifying the user action as corresponding to the fourth operation in the task, the system determines whether the fourth operation in the task can be performed before the second operation in the task. The system determines that the fourth operation should not be performed before the second operation (operation 418). For example, the system can compare the completion time associated with performing operations in a particular sequence with the threshold completion time of the task. The system determines that performing operation 4 before operation 2 results in a task completion time that exceeds the threshold. Based on determining that operation 4 should not be performed before operation 2 in the task, the system displays a notification on the display device 470 recommending that the user restart the task (operation 419). The system receives user input via the user interface that directs the system to allow operations of the task to be performed out of sequence (operation 430). The system determines (a) whether the user has authorization to reorder the steps in the task, and (b) whether the task can be completed in a different order than the stored sequence. For example, in a particular task being performed, some steps may be stored in a specific sequence, and other steps may be stored without any specific sequence. Thus, operation 4 may be stored with a dependency on operation 2, but operation 3 may be stored without any dependency. Therefore, operation 3 may be identified by the system as being capable of being performed in any sequence. Based on determining that: (a) the user has authorization to modify the dependencies in the task to change the sequence in which the task operations are performed; and (b) the task can be performed in the modified sequence, the system allows the user to reorder the sequence of operations to perform the task (operation 431). The system recalculates the estimated completion time for completing the task based on the reordered sequence (operation 432). The system can notify downstream processes of the updated estimated completion time, such as a workflow management system that manages the manufacture of components that use the particular component being manufactured by user 440.

[0112] refer to Figure 4F, the system can detect user actions even when no task has been selected by the user. The system detects the user action via video camera 428. The workspace monitoring platform 410 provides the video and sensor data to the user action classification machine learning model 411. Model 411 determines that the user action corresponds to operation 21 of the task ("place the component into the test machine 422") (operation 451). The system identifies a set of three tasks that include operation 21. The system further identifies the task that user 440 is most likely performing. For example, the three tasks that include operation 21 include operations that can be performed in any sequence. However, ML model 411 learns via training that the user only performed operation 21 at the beginning of the sequence of operations when performing task 3. Therefore, the system (a) determines a sequence of operations for displaying task 3, and (b) displays the user action associated with the next operation to be performed in task 3 (e.g., operation 22) via display device 470 (operation 452).

[0113] 6. Computer Networks and Cloud Networks

[0114] In one or more embodiments, the workspace monitoring system is implemented in a computer network. The computer network provides connectivity between a set of nodes. The nodes can be local and / or remote from each other. The nodes are connected via a set of links. Examples of links include coaxial cables, unshielded twisted pair cables, copper cables, optical fibers, and virtual links.

[0115] A subset of nodes implements a computer network. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses a computer network. Such nodes (also referred to as "hosts") can execute client processes and / or server processes. A client process requests a computing service (such as the execution of a specific application and / or the storage of a specific amount of data). The server process responds by performing the requested service and / or returning the corresponding data.

[0116] A computer network can be a physical network, comprising physical nodes connected by physical links. A physical node is any digital device. A physical node can be a hardware device with a specific function, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node can be a general-purpose machine configured to execute various virtual machines and / or applications that perform corresponding functions. A physical link is the physical medium that connects two or more physical nodes. Examples of links include coaxial cables, unshielded twisted-pair cables, copper cables, and optical fibers.

[0117] A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (such as a physical network). Each node in the overlay network corresponds to a corresponding node in the underlying network. Therefore, each node in the overlay network is associated with both an overlay address (for addressing the overlay node) and an underlying address (for addressing the underlying node that implements the overlay node). Overlay nodes may be digital devices and / or software processes (such as virtual machines, application instances, or threads). The links connecting overlay nodes are implemented as tunnels through the underlying network. The overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunnels are performed by encapsulation and decapsulation.

[0118] In an embodiment, a user can access the workspace monitoring platform via a client. The client can be local to the computer network and / or remote from the computer network. The client can access the computer network through other computer networks (such as a private network or the Internet). The client can use a communication protocol such as the Hypertext Transfer Protocol (HTTP) to pass requests to the computer network. The request is passed through an interface such as a client interface (such as a web browser), a program interface, or an application programming interface (API).

[0119] In an embodiment, a computer network provides connectivity between clients and network resources. Network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. Network resources are shared among multiple clients. Clients request computing services from the computer network independently of each other. Network resources are dynamically allocated to requests and / or clients on an on-demand basis. The network resources allocated to each request and / or client can be scaled based on, for example, (a) computing services requested by a specific client, (b) aggregated computing services requested by a specific tenant, and / or (c) the requested aggregated computing services of the computer network. Such a computer network can be referred to as a "cloud network."

[0120] In an embodiment, a service provider provides a cloud network to one or more end users. Various service models can be implemented through a cloud network, including but not limited to software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). In SaaS, the service provider provides end users with the ability to use the service provider's applications executed on network resources. In PaaS, the service provider provides end users with the ability to deploy customized applications on network resources. Custom applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider provides end users with the ability to supply processing, storage, network, and other basic computing resources provided by network resources. Any arbitrary application (including operating systems) can be deployed on network resources.

[0121] In embodiments, a computer network can implement various deployment models, including but not limited to private clouds, public clouds, and hybrid clouds. In a private cloud, network resources are provisioned for exclusive use by a specific group of one or more entities (the term "entity," as used herein, refers to a company, organization, individual, or other entity). Network resources can be located locally at the specific group of entities' premises and / or remotely from the specific group of entities' premises. In a public cloud, cloud resources are provisioned for multiple independent entities (also referred to as "tenants" or "clients"). A computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Several tenants can use the same specific network resources at different times and / or at the same time. Network resources can be located locally at the tenant's premises and / or remotely from the tenant's premises. In a hybrid cloud, the computer network includes both private and public clouds. The interface between the private and public clouds allows for data and application portability. Data stored in the private cloud and data stored in the public cloud can be exchanged via the interface. Applications implemented in the private cloud and applications implemented in the public cloud may have dependencies on each other. Calls from an application at the private cloud to an application at the public cloud (and vice versa) may be performed through the interface.

[0122] In an embodiment, the tenants of a multi-tenant computer network are independent of each other. For example, the business or operations of one tenant can be separated from the business or operations of another tenant. Different tenants may have different network requirements for the computer network. Examples of network requirements include processing speed, data storage capacity, security requirements, performance requirements, throughput requirements, latency requirements, resilience requirements, quality of service (QoS) requirements, tenant isolation and / or consistency. The same computer network may need to implement different network requirements required by different tenants.

[0123] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation methods can be used.

[0124] In an embodiment, each tenant is associated with a tenant ID. Each network resource of the multi-tenant computer network is labeled with a tenant ID. A tenant is permitted to access a specific network resource only if the tenant and the specific network resource are associated with the same tenant ID.

[0125] In an embodiment, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with the tenant ID. Additionally or alternatively, each data structure and / or dataset stored by the computer network is tagged with the tenant ID. A tenant is permitted to access a particular application, data structure, and / or dataset only if the tenant and the particular application, data structure, and / or dataset are associated with the same tenant ID.

[0126] As an example, each database implemented by a multi-tenant computer network can be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in a particular database. As another example, each entry in a database implemented by a multi-tenant computer network can be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in a particular entry. However, a database can be shared by multiple tenants.

[0127] In one embodiment, a subscription list indicates which tenants have authorization to access which applications. For each application, a list of tenant IDs of tenants authorized to access that application is stored. A tenant is granted access to a particular application only if its tenant ID is included in the subscription list corresponding to the particular application.

[0128] In an embodiment, network resources corresponding to different tenants (such as digital devices, virtual machines, application instances, and threads) are isolated from tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, packets from any source device in a tenant overlay network can only be transmitted to other devices within the same tenant overlay network. Encapsulation tunnels are used to prohibit any transmission from a source device on a tenant overlay network to devices in other tenant overlay networks. Specifically, a packet received from a source device is encapsulated within an outer packet. The outer packet is transmitted from a first encapsulation tunnel endpoint (communicating with a source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet transmitted by the source device. The original packet is transmitted from the second encapsulation tunnel endpoint to a destination device in the same specific overlay network.

[0129] 7. Other; Extension

[0130] Embodiments are directed to a system having one or more devices including a hardware processor and configured to perform any of the operations described herein and / or recited in any of the following claims.

[0131] In an embodiment, a non-transitory computer-readable storage medium comprises instructions that, when executed by one or more hardware processors, cause performance of any of the operations described herein and / or recited in any of the claims.

[0132] Any combination of the features and functions described herein may be used in accordance with one or more embodiments. In the foregoing description, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what the applicants intend to be the scope of the invention, is the literal and equivalent scope of the set of claims claimed in this application, in the specific form in which such claims are claimed, including any subsequent corrections.

[0133] 8. Hardware Overview

[0134] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the technology, or can include a digital electronic device (such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs)) that is permanently programmed to perform the technology, or can include one or more general-purpose hardware processors that are programmed to perform the technology according to program instructions in firmware, memory, other storage devices, or a combination. Such special-purpose computing devices can also combine customized hard-wired logic, ASICs, FPGAs, or NPUs with customized programming to complete the technology. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that incorporates hard-wiring and / or program logic to implement the technology.

[0135] For example, Figure 5 5 is a block diagram illustrating a computer system 500 upon which embodiments of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0136] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 to store information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible to processor 504, render computer system 500 into a special-purpose machine customized to perform the operations specified in the instructions.

[0137] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0138] The computer system 500 may be coupled to a display 512, such as a cathode ray tube (CRT), via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 504 and for controlling cursor movement on the display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0139] The computer system 500 can implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic (which, in combination with the computer system, makes the computer system 500 a special-purpose machine or programs it as a special-purpose machine). According to one embodiment, the techniques herein are performed by the computer system 500 in response to the processor 504 executing one or more sequences of one or more instructions contained in the main memory 506. Such instructions can be read into the main memory 506 from another storage medium (such as the storage device 510). Execution of the sequences of instructions contained in the main memory 506 causes the processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

[0140] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chip or tape cartridge, content addressable memory (CAM), and ternary content addressable memory (TCAM).

[0141] Storage media are distinct from, but can be used in conjunction with, transmission media. Transmission media are involved in transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and fiber optics, including the lines comprising bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0142] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into the remote computer's dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal and appropriate circuitry may place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on the storage device 510 before or after execution by the processor 504.

[0143] Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides two-way data communication coupled to a network link 520 connected to a local network 522. For example, communication interface 518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, communication interface 518 can be a LAN card that provides a data communication connection to a compatible local area network (LAN). Wireless links can also be implemented. In any such implementation, communication interface 518 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0144] The network link 520 typically provides data communication to other data devices through one or more networks. For example, the network link 520 can provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an Internet service provider (ISP) 526. The ISP 526, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 528. Both the local network 522 and the Internet 528 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518, which carry the digital data to and from the computer system 500, are example forms of transmission media.

[0145] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520, and communication interface 518. In the Internet example, server 530 can transmit the requested code for an application program through Internet 528, ISP 526, local network 522, and communication interface 518.

[0146] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510 or other non-volatile storage for later execution.

[0147] In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what the applicants intend to be the scope of the invention, is the literal and equivalent scope of the set of claims claimed in this application, in the specific form in which such claims are claimed, including any subsequent corrections.

Claims

1. A non-transitory computer-readable medium comprising instructions that, when executed by one or more hardware processors, cause operations comprising: Presenting a first instruction associated with a first operation in a set of operations for the user to perform the first operation; concurrently with presenting the first instruction: analyzing the video stream in real time as the video stream is received to detect a first set of one or more actions performed by the user; determining whether the first set of actions corresponds to completion of a first operation associated with a first instruction currently being presented; as well as In response to determining that the first set of actions corresponds to completion of the first operation: A second instruction corresponding to a second operation subsequent to the first operation in the set of operations is presented, for the user to perform the second operation.

2. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: concurrently with presenting the second instruction: analyzing the video stream in real time as the video stream is received to detect a second set of actions among the one or more actions performed by the user; determining whether the second set of actions corresponds to a second operation associated with a second instruction currently being presented; as well as In response to determining that the second set of actions corresponds to the second operation: It is determined that the user has started to perform a second operation.

3. The non-transitory computer-readable medium of claim 2, wherein the operations further comprise: In response to determining that the user has begun performing the second operation, third instructions associated with a third operation in the set of operations are presented to perform the third operation.

4. The non-transitory computer-readable medium of claim 2, wherein the operations further comprise: In response to determining that the second set of actions corresponds to the second operation: A start time of execution of the second operation is detected.

5. The non-transitory computer-readable medium of claim 4, wherein the operations further comprise: Scheduling-related information is determined based on the start time of execution of the second operation.

6. The non-transitory computer-readable medium of claim 5, wherein determining the scheduling-related information comprises at least one of: Calculating future delays in the workflow, planning resource ramp-ups for specific times, and predicting completion times for one or both of a second operation and a task including the second operation.

7. The non-transitory computer-readable medium of claim 4, wherein the operations further comprise: detecting an end time of execution of the second operation; as well as A total time to complete the operation is calculated based on the start time and the end time.

8. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: concurrently with presenting the second instruction: analyzing the video stream in real time as the video stream is received to detect a second set of actions among the one or more actions performed by the user; determining whether the second set of actions corresponds to completion of a second operation associated with a second instruction currently being presented; as well as In response to determining that the second set of actions corresponds to a third operation different from the second operation: Determining that the second set of actions is out of sequence; and The notification is presented out of sequence based on the second set of actions.

9. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: concurrently with presenting the second instruction: analyzing the video stream in real time as the video stream is received to detect a second set of actions among the one or more actions performed by the user; determining whether the second set of actions corresponds to completion of a second operation associated with a second instruction currently being presented; as well as In response to determining that the second set of actions corresponds to a third operation different from the second operation in the set of operations: The sequence of the set of operations is modified to present instructions corresponding to the third operation before presenting instructions corresponding to the second operation.

10. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: Identify the tasks to be completed; as well as The set of operations is determined based on the task to be completed.

11. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: analyzing the video stream in real time as the video stream is received to detect performance of a specific action; as well as In response to detecting the specific action, an operation is performed to capture system data.

12. The non-transitory computer-readable medium of claim 1 , wherein a sequence for performing a set of operations including a first operation is generated by a machine learning model based on a task completion metric associated with performing the set of operations in an order corresponding to the sequence.

13. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: identifying a task including a first operation, a second operation, and a third operation sequenced subsequent to the second operation; concurrently with presenting the second instruction: analyzing the video stream in real time as the video stream is received to detect a second set of actions among the one or more actions performed by the user; determining whether the second set of actions corresponds to completion of a second operation associated with a second instruction currently being presented; determining that the second set of actions corresponds to the triggering criteria; Based on determining that the second set of actions corresponds to the triggering criteria, performing a triggering operation, including at least one of the following: Presenting different remedial instructions than the third operation; and A prompt is presented to receive user input regarding modifying the task to include a fourth operation, the fourth operation including the second set of actions.

14. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: concurrently with presenting the second instruction: analyzing the video stream in real time as the video stream is received to detect a second set of actions among the one or more actions performed by the user; determining whether the second set of actions corresponds to completion of a second operation associated with a second instruction currently being presented; as well as In response to determining that: (a) the second set of actions corresponds to completion of the second operation, and (b) the second set of actions does not correspond to actions specified in the second instruction: The second instruction is modified to include a second set of actions.

15. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: In response to determining that the first set of actions corresponds to said completion of the first operation: A task profile associated with the set of operations is updated to indicate that the first operation has been completed.

16. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: calculating estimated completion times for different orders of performing the set of operations; as well as Based on the estimated completion time, a particular order is selected to present the instructions corresponding to the set of operations.

17. The non-transitory computer-readable medium of claim 1, wherein the set of operations specifies a sequence to order operations in the set of operations, and wherein the second operation is determined based on the sequence.

18. The non-transitory computer-readable medium of claim 1, wherein presenting the second instructions is performed in the absence of a user input requesting a switch from presenting the first instructions to presenting the second instructions.

19. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise: In response to determining that the first set of actions corresponds to said completion of the first operation: A value representing at least one component consumed during execution of the first operation is updated in the data store.

20. The non-transitory computer-readable medium of claim 1, wherein the operations further comprise: In response to determining that the first set of actions corresponds to said completion of the first operation: An amount of resources that can be consumed by performing the set of operations in subsequent instances of the set of operations is detected by at least one sensor.

21. A method comprising the operations of any one of claims 1 to 20.

22. A system comprising one or more processors; and A memory storing instructions, which, when executed by the one or more processors, cause the system to perform the operations of any one of claims 1 to 20.

23. A system comprising means for performing the operations of any one of claims 1 to 20.

Images