Assembly Monitoring System

Abstract

A system and method for receiving, using one or more processors, image data, the image data including a first training video depicting the performance of one or more steps on a first workpiece; applying, using the one or more processors, a first set of labels to the first training video based on user input; performing, using the one or more processors, extraction on the image data to thereby generate extracted information, the extracted information including first extracted image information related to the first training video; and training, using the one or more processors, a process monitoring algorithm based on the extracted information and the first set of labels.

Description

[Background technology]

[0001] The present disclosure relates to systems and methods for monitoring one or more processes. The ability to train workers to accurately and efficiently repeat a process and then be able to repeat that process accurately and efficiently is important to various industries and verticals. Currently, assembly processes are documented using standard job instruction sheets (JIS). JIS is how most modern manufacturers document their assembly operations. Typically, these JIS documents reside at the work station to remind operators as needed, and new operators are trained using the JIS. However, the JIS system is outdated and has limitations. Summary of the Invention [Means for solving the problem]

[0002] In general, innovative aspects of the subject matter described in this disclosure may be embodied in a method that includes: receiving, using one or more processors, image data, where the image data includes a first training video depicting the performance of one or more steps on a first workpiece; applying, using the one or more processors, a first set of labels to the first training video based on user input; performing, using the one or more processors, extraction on the image data, thereby generating extracted information, where the extracted information includes first extracted image information related to the first training video; and training, using the one or more processors, a process monitoring algorithm based on the extracted information and the first set of labels.

[0003] Other implementations of one or more of these aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods encoded on computer storage devices. These and other implementations may optionally include one or more of the additional features, respectively.

[0004] These and other implementations may optionally include one or more of the following features, respectively. For example, applying a first set of labels to the first training video based on user input may further include receiving a plurality of sets of bounding boxes from the user, each set of bounding boxes associated with a single workpiece, and the first set of labels including the first set of bounding boxes; training an object detection algorithm based on the plurality of sets of bounding boxes, wherein extracting includes applying the object detection algorithm, and the extracted image information includes the object detection. For example, the first set of bounding boxes may include a first bounding box around the first workpiece in a start frame, a second bounding box around the first workpiece in an intermediate frame, and a third bounding box around the first workpiece in an end frame, and training the object detection algorithm includes applying a deal learning object detection network. For example, the first set of bounding boxes may represent a first workpiece at the beginning, middle, and end of the cycle within the field of view of the camera device. For example, applying the first set of labels to the first training video based on user input may further include receiving a set of polygons from the user, where the set of polygons includes one or more of a pick area polygon, a walking area polygon, and a pitch polygon, and the first set of labels includes the set of polygons. For example, performing extraction on the image data may generate one or more of pose data and object detection. For example, the set of labels may include a start frame label associated with a frame where the step starts, an end frame label associated with a frame where the step ends, and a step identifier that identifies the step.For example, training the process monitoring algorithm may be further based on the second extracted information and the second set of labels. For example, the method may include, after training the process monitoring algorithm, receiving live video image data, performing extraction on the live video image data to thereby generate extracted information of the live video, applying the process monitoring algorithm to the live video using the extracted information of the live video, and presenting step data. For example, performing extraction on the live video image data may include obtaining pose data and object detection, and applying the process monitoring algorithm and presenting the step data may include applying the process monitoring algorithm to extracted data of a first live video associated with a first time to generate a first view and presenting first step data including the first view, where the first view includes a step identifier and whether the step is complete; applying the process monitoring algorithm to extracted data of a second live video associated with a second time to generate a second view by updating the first view; and presenting second step data including the second view.

[0005] It is to be understood that this list of features and advantages is not exhaustive and that many additional features and advantages are contemplated and are within the scope of the present disclosure. It is further to be understood that the language used in this disclosure has been selected primarily for ease of reading and didactic purposes, and not to limit the scope of the subject matter disclosed herein.

[0006] The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numerals are used to refer to similar elements and in which: [Brief explanation of the drawings]

[0007] [Figure 1]FIG. 1 is a block diagram of an exemplary implementation of a system for assembly monitoring, according to one implementation. [Figure 2] FIG. 1 is a block diagram of an exemplary implementation of a camera device for assembly monitoring, according to one implementation. [Figure 3] FIG. 1 is an exemplary diagram of a standard operating manual. [Figure 4] FIG. 1 is an exemplary block diagram illustrating a training process according to one implementation. [Figure 5] FIG. 10 is an exemplary diagram illustrating normalization and shifting for a movement process, according to one implementation. [Figure 6] 1 is a flow diagram of an exemplary method for generating a process monitoring algorithm, according to one implementation. [Figure 7] 1 is a flow diagram of an exemplary method for monitoring a process, according to one implementation. [Figure 8] 10 is a flow diagram illustrating another exemplary method of monitoring a process, according to one implementation. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present disclosure relates to systems and methods for monitoring assembly processes. Monitoring assembly processes involves many challenges and has several unmet needs. First, current mechanisms, such as the JIS, do not provide real-time feedback to indicate when steps are skipped or performed out of sequence. Second, current systems lack effective tracking mechanisms for identifying the root cause of assembly errors. For example, current systems require humans to comb through large amounts of video (if video is available) to attempt to determine who may have worked on the defective unit and how the error occurred. Third, current systems of assembly monitoring do not highlight optimizations that might be made to improve assembly efficiency or provide other benefits (e.g., reducing repetitive stress or injuries or maintaining social distancing).

[0009] Using machine learning to address the above-mentioned shortcomings presents several technical challenges. First, there is the cold start problem. Existing machine learning mechanisms require a large number of examples on which algorithms may be trained and tested. Collecting and labeling a sufficient amount of usable example data (when using supervised machine learning) is difficult. This is particularly problematic for assembly processes that may change frequently for any number of reasons, such as right-handed vs. left-handed workers, sometimes referred to herein as "operators," relocation or reorientation of tools, parts, or work stations as the factory floor is rearranged, or changes in products on the same assembly line (e.g., 2WD transmissions vs. 4WD transmissions). Second, the computationally intensive bandwidth utilization of generating and applying machine learning may make applying machine learning impractical.

[0010] The technology introduced herein at least partially overcomes the shortcomings and limitations of the prior art by providing a system and method for monitoring an assembly process. It is to be understood that the language used in this disclosure has been chosen primarily for ease of reading and didactic purposes, and not to limit the scope of the subject matter disclosed herein.

[0011] The present disclosure describes systems and methods for monitoring processes. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it should be noted that the present disclosure may be practiced without these specific details. For example, while the monitored process may be referred to as an "assembly process," the workpiece may be being assembled, disassembled, or modified. As another example, while the examples herein may refer to factory assembly of workpieces that are products (e.g., automobile transmissions), it should be understood that the monitored workpieces and processes are varied and may include, by way of example and not limitation, sandwiches in a fast food restaurant kitchen, anatomical tissue undergoing routine medical procedures, routine maintenance or inspection processes of machinery, etc.

[0012] 1 is a block diagram of an exemplary system 100 for monitoring a process, according to one implementation. In the exemplary implementation shown, system 100 includes a cloud or network 150, camera devices 152a-n, a client device 156, and a server 162. Although not shown, other IoT devices may also be coupled to network 150 for interaction. Camera device 152a / n is described in more detail below with reference to FIG. 2.

[0013] Network 150 may be of a conventional type, wired, or wireless, and may have many different configurations, including star, token ring, or other configurations. Furthermore, network 150 may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), and / or other interconnected data paths through which multiple devices may communicate. In some implementations, network 150 may be a peer-to-peer network. Network 150 may also be coupled to or include part of a telecommunications network for transmitting data in a variety of different communication protocols. In some other implementations, network 150 includes a Bluetooth or cellular communication network for transmitting and receiving data via Short Messaging Service (SMS), Multimedia Messaging Service (MMS), Hypertext Transfer Protocol (HTTP), direct data connection, Wireless Access Protocol (WAP), email, etc. Furthermore, while FIG. 1 shows a single network 150 coupled to camera devices 152a-n, client device 156, and server 162, in practice, one or more networks 150 may be connected to these entities.

[0014] The client device 156 of FIG. 1 is used as an example. While one client device 156 is shown, the present disclosure applies to a system architecture having any number of client devices available to any number of users (not shown). In the illustrated implementation, the client device 156 is communicatively coupled to the network 150 via respective signal lines. Furthermore, it should be understood that there may be more, fewer, or different client devices than those shown in FIG. 1 . For example, the client device 156 may be any computing device including memory and a processor. For example, the client device 156 could be a laptop computer, a desktop computer, a tablet computer, a mobile phone, a smartphone, a personal digital assistant, a mobile email device, a portable game player, a portable music player, a television with one or more processors embedded or coupled thereto, or any other electronic device capable of accessing the network 150.

[0015] The client device 156 may include one or more of an application (not shown) or a web browser (not shown). In some implementations, a user (not shown) is presented with a user interface associated with the assembly monitoring module 203 described below with reference to FIG. 2 and provides user input (e.g., labels) by interacting with the app or web browser on the client device 156, at least some of which user input is provided to the assembly monitoring module 203 (or subcomponents thereof) to provide the functionality described herein.

[0016] A server 162 is shown by way of example. Although a single server 162 is shown, the present disclosure applies to system architectures having one or more servers 162, either distributed (e.g., cloud) or local (virtual or physical).

[0017] Camera devices 152a and 152n are shown as examples. Although two camera devices are shown, this description applies to any architecture having one or more camera devices 152.

[0018] 2 is a block diagram of an exemplary camera device 152. As shown, camera device 152 may include a communication unit 202, a processor 204, a memory 206, a database 208, an input device 212, an output device 214, and an assembly monitoring module 203, which may be communicatively coupled by a communication bus 210.

[0019] 2 is provided as an example and may take other forms and include additional or fewer components without departing from the scope of the present disclosure. For example, the various components of camera device 152 may be coupled for communication using various communication protocols and / or technologies, including, for example, a communication bus, a software communication mechanism, a computer network, etc. Although not shown, camera device 152 may include various operating systems, sensors, additional processors, and other physical configurations. Communication unit 202, processor 204, memory 206, etc. represent one or more of these components.

[0020] The communications unit 202 may include one or more interface devices (I / F) for wired and wireless connections between components of the system 100. For example, the communications unit 202 may include various types of known connections and interface options, without limitation. The communications unit 202 may be coupled to other components of the server 162 via a bus 210. The communications unit 202 may provide other connections to the network 150 and to other entities of the system 100 using various standard communications protocols.

[0021] In some implementations, the communication unit 202 includes a wired transceiver for exchanging data with other entities, hi some implementations, the communication unit 202 includes a wireless transceiver for exchanging data with other entities, or any other communication channel using one or more wireless communication methods, such as IEEE 802.11, IEEE 802.16, BLUETOOTH, or other suitable wireless communication methods.

[0022] In some implementations, the communication unit 202 includes a cellular communication transceiver for transmitting and receiving data over a cellular communication network, such as via Short Messaging Service (SMS), Multimedia Messaging Service (MMS), Hypertext Transfer Protocol (HTTP), direct data connection, WAP, email, or other suitable types of electronic communication. In some implementations, the communication unit 202 includes a wired port (e.g., CAT5 or 6, USB, Fiber, etc.) and a wireless transceiver. The communication unit 202 also provides other conventional connections to networks for delivery of files and / or media objects using standard network protocols such as TCP / IP, HTTP, HTTPS, and SMTP, as will be appreciated by those skilled in the art.

[0023] The processor 204 may include an arithmetic logic unit, a microprocessor, a general-purpose controller, or some other processor array for performing calculations and providing electronic display signals to a display device. The processor 204 may be coupled to a bus 210 for communication with other components. The processor 204 processes data signals and may include various computing architectures, including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture implementing a combination of instruction sets. Although only a single processor is shown in FIG. 2, multiple processors may be included. The processing power may be sufficient to perform complex tasks, including various types of feature extraction and matching. It should be apparent to those skilled in the art that other processors, operating systems, sensors, displays, and physical configurations are possible.

[0024] The memory 206 stores instructions and / or data that may be executed by the processor 204. The memory 206 is coupled to the bus 210 for communication with other components. The instructions and / or data may include code for performing any and / or all of the techniques described herein. The memory 206 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory, or some other memory device known in the art. In some implementations, the memory 206 also includes non-volatile memory or similar persistent storage devices and media, such as a hard disk drive, a floppy disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device known in the art for more permanently storing information. Although shown separately, the assembly monitor module 203 may be stored in the memory 206 and, in some implementations, cause the processor 204 to perform its functions.

[0025] Database 208 may include information sources for storing and providing access to data that may be used by assembly monitoring module 203. For example, database 208 may store one or more of video data (e.g., MP4, MPEG, AVI, etc.), video extraction data (e.g., pose data), labels (e.g., start and end frames), data obtained from a manufacturer's data acquisition and storage system (not shown), etc.

[0026] In some implementations, database 208 may store data associated with a database management system (DBMS) operable on server 162. For example, the DBMS may include a Structured Query Language (SQL) DBMS, a NoSQL DBMS, various combinations thereof, etc. In some cases, the DBMS may store data in multidimensional tables consisting of rows and columns and manipulate (e.g., insert, query, update, and / or delete) rows of data using programmatic operations.

[0027] Database 208 may be included in camera device 152 or may be included in another computing and / or storage system distinct from camera device 152 but coupled to or accessible by camera device 152. For example, in some implementations, database 208 may be network-attached storage (not shown) communicatively coupled to camera device 152 via network 150 or may be accessed via a third-party system. Database 208 may include one or more non-transitory computer-readable media for storing data. In some implementations, database 208 may be merged with or distinct from memory 206.

[0028] The input device 212 includes an image / video capture device (e.g., a camera). In some implementations, the input device 212 may include one or more peripheral devices. For example, the input device 212 may include a camera sensor, a keyboard, a pointing device, a microphone, a touchscreen display integrated with the output device 214, etc.

[0029] The output device 214 may be any device capable of outputting information. The output device 214 may include one or more of a display (LCD, OLED, etc.), a printer, a tactile device, an audio playback device, a touchscreen display, a remote computing device, etc. In some implementations, the output device is a display that may display electronic images and data output by the processor 204 for presentation to a user.

[0030] Bus 210 may include a communications bus for transferring data between components of or between computing devices, a network bus system including network 150 or portions thereof, a processor mesh, combinations thereof, etc. Software communication mechanisms may include and / or facilitate, for example, inter-method communications, local function or procedure calls, remote procedure calls, object brokers (e.g., CORBA), direct socket communications between software modules (e.g., TCP / IP sockets), UDP broadcast and receive, HTTP connections, etc. Additionally, any or all of the communications may be secure (e.g., SSH, HTTPS, etc.).

[0031] Components 202, 203, 204, 206, 208, 212, and / or 214 may be communicatively coupled with each other and / or other components of server 162 by bus 210 and / or processor 204. In some implementations, components 202, 203, 204, 206, 208, 212, and / or 214 may include computer logic (e.g., software logic, hardware logic, etc.) executable by processor 204 to provide the actions and / or functionality described herein. In any of the above implementations, these components 202, 203, 204, 206, 208, 212, and 214 may be adapted to cooperate and communicate with processor 204 and other components of camera device 152.

[0032] 2, assembly monitoring module 203 may include image extraction module 209, labeling module 211, process monitoring module 213, tracking module 217, and process optimization module 219. These modules 209, 211, 213, 217, and 219 provide the functionality described below. Note that system 100 of FIG. 1 and camera device 152 of FIG. 2 are examples, and architectures may vary, and components, features, or functions may be moved or reconfigured, and such modifications are within the scope of this disclosure. For example, in some implementations, tracking module 217 and / or processing module 219 may be server-based and located on server 162.

[0033] Currently, assembly processes are documented using standard operating instructions (JIS), an example of which is shown in Figure 3. JIS is how most modern manufacturers document their assembly operations. Typically, these JIS documents are at the work site to remind operators as needed, and new operators are trained using the JIS. To summarize, and to simplify, the JIS is now the ground truth for assembly. However, the assembly monitoring system described herein replaces the older system of standard operating instructions.

[0034] In some implementations, the assembly monitoring module 203 includes an image extraction module 209 , a labeling module 211 , a process monitor module 213 , a tracking module 217 , and a process optimization module 219 .

[0035] The features and functions of the above-mentioned modules of the assembly monitoring module 203 are described in detail below. However, a rough mapping between modules 209, 211, 213, 217, and 219 and their respective roles is as follows: Image extraction module 209 and labeling module 211 are used to preprocess data, e.g., create pose data, detect objects, and apply labels, respectively. The preprocessed data is used by process training monitor 213 to train algorithms, which may then test and deploy them. For example, the preprocessed data is provided to process monitor module 213, which trains and deploys AI that may identify which steps are being performed and whether steps are being skipped and alert the assembler (e.g., alert the assembler of skipped steps, steps performed out of sequence, steps performed incorrectly, etc. via an application (not shown) on the client device 156 at the assembler's workstation). The tracking module 217 and process optimization module 219 perform post-processing. For example, the tracking module 217 provides traceability and visibility into the process that allows the root cause of a mistake to be quickly and accurately identified, and the optimization module 219 may provide suggestions to those performing the assembly or another party (e.g., management) to further optimize and streamline the assembly.

[0036] The image extraction module 209 receives image data (e.g., video) and performs image extraction (e.g., to generate pose data or detect objects from the video). Examples of extraction include, but are not limited to, pose estimation and object detection.

[0037] In some implementations, image extraction includes object detection. For example, the image extraction module 209 detects whether a tool is present in an image (e.g., a video frame). Examples of tools include, by way of example and not limitation, drills, wrenches, screwdrivers, hammers, punches, pliers, etc. Detecting objects may enable discrimination of different steps using different tools. In some implementations, tracking objects may provide path information similar to pose estimation.

[0038] In some implementations, the image extraction module 209 trains and applies a Deal learning object detection network to perform object detection, although other mechanisms for object detection exist and may be used depending on the implementation.

[0039] In some implementations, the object detection is trained based on a multi-bounding box approach that uses several bounding boxes, for example, between two and ten, including two and ten, per instance. For example, in some implementations, a video of the process is presented during object detection training, and a user uses the labeling module 211 (described below) to identify three bounding boxes for training (i.e., a first bounding box around the workpiece in the starting frame, a second bounding box around the workpiece in a frame approximately halfway through the execution of the step / cycle, and a third bounding box around the workpiece in the ending frame), and in some implementations, the image extraction module 209 obtains multiple sets of three bounding boxes as training data from the user labeling videos of multiple instances of the process and applies the Deal learning object detection network to the sets of three bounding boxes without any further labels. As used herein, a step may refer to a specific action (e.g., drill a hole in bolt 1), and a cycle refers to that assembly location / portion of an assembly line / set of actions that are to take place within the field of view of camera device 152. In some implementations, a cycle may represent an entire process (i.e., all steps of a process), or a subset of steps of a larger process.

[0040] It should be appreciated that by using the multi-bounding box approach and the Deal learning object detection network, object detection may be accurate and reliable using orders of magnitude less training data. For example, the systems and methods described herein may use training data describing, for example, a dozen instances of a workpiece going through cycles / steps, while alternative methods may require thousands of instances to train and produce reliable results.

[0041] In some implementations, image extraction includes applying pose estimation to image data (e.g., video). In some implementations, the pose estimation generated by image extraction module 209 takes individual image frames and outputs pose data as an intermediate representation for either statistical or other deep learning methods (depending on the implementation). Tracking pose data may allow system 100 to gather information about which step is currently being performed.

[0042] In some implementations, the image extraction module 209 generates a pose estimate for each image it receives (e.g., for consecutive video frames). In some implementations, the image extraction module 209 generates a whole-body pose for an image that includes a human. For example, the image extraction module 209 generates the whole-body pose based on 17 keypoints.

[0043] In some implementations, the image extraction module 209 trains and applies a pose estimation neural network to perform pose estimation, although other mechanisms for pose estimation exist and are within the scope of this description.

[0044] It should be appreciated that in the illustrated embodiment, the image extraction module 209 is shown within the camera device. While in some implementations the pose estimation performed by the image extraction module 209 may be performed on a server or other hardware device, it should be appreciated that the image extraction module 209 described herein may, according to some implementations, be efficient enough to run on the camera device 152 itself, despite the camera device's limited hardware capabilities (e.g., processing power, memory, etc.). Such an implementation may provide many advantages, especially when a large number of cameras are installed, as is typical in large assembly facilities. The advantages may include reduced bandwidth requirements (because cameras do not need to send data to a server for pose estimation), reduced costs (because fewer server and bandwidth resources need to be installed or made available), reduced latency (because pose estimation is performed locally and distributed among multiple cameras), and the ability to use multiple cameras to cover a process, cycle, or step from multiple angles.

[0045] In some implementations, multiple camera devices 152 in system 100 may cooperate and communicate. For example, in some implementations, multiple camera devices 152 cooperate within system 100 to identify a step or missed step despite one camera being unable to see the entire step or process (e.g., because one camera is occluded or has an obstructed view).

[0046] The labeling module 211 generates a workflow and a set of labels. In some implementations, the labeling module 211 defines workflow steps based on physical endpoints. For example, in some implementations, the labeling module 211 defines the steps "grab bolt," "drive bolt 1," "drive bolt 2," and "drive bolt 3" because each of these actions has a physical endpoint (e.g., where the bolt rests after being driven into place). In contrast, the JIS might have defined "grab and drive bolts 1-3" as a single step.

[0047] In some implementations, the labeling module 211 labels a start frame, an end frame, a step identifier (e.g., a name or number), and active keypoints. In some implementations, the keypoints are a subset of keypoints from those used by the image extraction module 209 when performing pose estimation. Because humans often assemble things by hand, keypoints associated with the hand (e.g., one or more fingers or wrist) or arm (e.g., elbow) are often labeled as active keypoints when humans perform assembly. However, other active keypoints may be identified (e.g., pivot points for a robotic arm, or feet for a foot-actuated / operated device).

[0048] By labeling the start and end frames of a step, the process monitoring module 213, discussed further below, may derive one or more of a set path, end point, movement trajectory, or other behavioral interpretation of the tool being used or the process being performed. Additionally, the start and end frame labels indirectly represent the time taken to complete the step.

[0049] In some implementations, the labeling module 211 labels the pose (e.g., position and orientation) of the human. Depending on the implementation and / or use case, the labeled pose may be for one of the limbs, a set of limbs, or the whole body. Labeling the human pose generates a correlation between steps and the human pose.

[0050] In some implementations, the labeling module 211 labels the pose (e.g., position and orientation) of the tool. The labeling of the tool pose generates a correlation between the step and the tool pose.

[0051] In some implementations, the labeling module 211 labels example steps with categories. In some implementations, the categories may correspond to how well and / or how quickly the step was performed. For example, the labeling module categorizes steps as ideal, abnormal, satisfactory, or very poor, or alternatively, good, poor, or fair, to create a benchmark for process performance determination applications. In some implementations, a step may be labeled as "critical," and such a step may generate an alert when missed or skipped.

[0052] In some implementations, the labeling module 211 may define a bounding box around a worker or work station to reduce noise and background motion and increase the accuracy of the process monitor module 213 or its algorithms.

[0053] While many of the examples discussed herein are step-level labels, it should be recognized that cycle-level labels are within the scope of this disclosure. For example, rather than or in addition to labeling a step (e.g., installing a bolt in a transmission) with a category such as "good," a set of steps may be assigned a label (e.g., the entire assembly of a transmission may be labeled "good").

[0054] Labeling examples by category based on speed and accuracy of step execution may advantageously allow the process optimization module 219, discussed below, to analyze the categories for commonalities and provide suggestions. For example, the optimization module 219 may analyze "ideal" examples and suggest optimized tool placement or movement profiles, part placement and orientation, etc., to improve efficiency or reduce the risk of repetitive stress injuries. As another example, the optimization module 219 may analyze "very poor" examples and suggest actions, such as retraining a particular poor performer because bolt driving at a station is taking longer than expected, inspecting a particular tool for wear / breakage, or inspecting assembly parts to ensure they meet specifications.

[0055] In some implementations, the labeling module 211 generates a subset of labels based on manual user input and automatically generates a subset of labels. In some implementations, the labeling module 211 generates a subset of labels based on manual user-input labeling (e.g., via a client device) of a set of training image data during a training period. The user-input labels obtained by the labeling module 211 may vary based on the implementation and / or use case (e.g., a moving process or a stationary process). Examples of labels that the labeling module 211 may prompt and receive from the user may include, but are not limited to, one or more of a start frame, an end frame, a region of interest, a set of keypoints, an identifier, etc.

[0056] In some implementations, the labeling module 211 receives user input identifying a step, for example, by name or number. For example, in some implementations, the labeling module 211 may present several records in which a worker performed one or more steps, and for each of the records, receives from the user a label identifying the step (e.g., "Drive bolt 1," "Drive bolt 2," etc.).

[0057] In some implementations, the labeling module 211 receives user input identifying a start frame and an end frame of an example instance (or set of example instances) of a step recorded by the camera device 152. For example, in some implementations, the labeling module 211 may present several recordings of a worker performing one or more steps, and for each of the recordings, receives a label from the user identifying the start frame (e.g., the frame at which the worker begins driving bolt 1) and the end frame (e.g., the frame at which bolt 1 is secured).

[0058] In some implementations, the labeling module 211 receives user input identifying a set of key points of interest (e.g., the right wrist, the left wrist, or both wrists) in an example instance (or set of example instances) of a step recorded by the camera device 152.

[0059] In some implementations, the labeling module 211 may receive from a user user-inputted labels that identify one or more regions of interest. Examples of regions of interest may include, but are not limited to, a bounding box around a workpiece, a pick area (i.e., an area where parts or components to be picked up / removed are stored), a walking area (i.e., an area where a monitored worker may walk (e.g., between the workpiece and the pick area)), and a pitch. For example, in some implementations, the labeling module 211 may receive from a user identification of a first bounding box around a workpiece (e.g., in a start frame), a second bounding box around a workpiece (e.g., in a frame halfway through the execution of a step / cycle), and a third bounding box around a workpiece (e.g., in an end frame). In another example, in some implementations, the labeling module 211 may receive from the user the identification of a first polygon identifying a pick area (e.g., where the user picks up the bolt to be driven) and a second polygon identifying a walk area (e.g., the area between where the worker picks up the bolt and where the worker drives the bolt). It should be appreciated that the use of a “walk area” label may reduce problems caused by other people or objects that appear within the camera's field of view but outside of that area, and may also be used to distinguish worker movement when the worker is on the conveyor and not actually walking. In yet another example, in some implementations, the labeling module 211 may receive from the user the identification of a pitch (e.g., pitch start and pitch end labels). A moving assembly line is generally divided into several stations (e.g., in automobile manufacturing, an assembly line is expected to be 6-10 feet long on average, but this may vary based on the size of the product and the speed at which the line moves). A pitch is where a set of steps related to the moving process is to be performed.

[0060] In some implementations, labeling based on manual user input may occur before training and deployment, and the labeling module 211 may automatically label future instances of the step after training and deployment. For example, after training, a factory worker drives bolt 1, and the labeling module 211 automatically generates the label "drives bolt 1" and labels the factory worker's performance in that instance as "ideal."

[0061] In some implementations, the labeling module 211 automatically generates a subset of labels. For example, when the first instance of a process cycle has already been manually labeled by a user (e.g., indicating the start and end frames of each step, step identifiers, and key points of interest), in some implementations, the labeling module 211 automatically labels the second training instance to expedite the labeling process during training, and the user may (optionally) review and revise the automatically assigned labels when corrections are needed. It should be appreciated that this may alleviate the cold start problem and may rapidly create a large number of labeled sets for training in less time and with little human intervention. In another example, in some implementations, the labeling module 211 automatically generates a run of a factory cycle after a process monitoring algorithm is deployed and labels it as “ideal” because all steps have been executed and a time threshold (e.g., a threshold based on the average or median number of seconds to execute the cycle) has been met.

[0062] In some implementations, the labels generated by the labeling module 211 are used by the image extraction module 209 to train one or more of a pose estimator and an object detection. In some implementations, the labels generated by the labeling module 211 are used by the process monitor module 213 to train the process monitoring module 213. In some implementations, labels associated with videos received during deployment are not used by the process monitoring algorithms, while in some implementations, labels automatically generated by the labeling module 211 and applied to videos received during deployment are used by one or more of the tracking module 217 and the process monitoring module 219. For example, the automatically generated labels may be used to determine the time spent on the workpiece by the worker in a cycle, the time spent in the pick area per cycle also known as “pick time,” the time spent by the operator walking in the cycle also known as “walk time,” the cycle time (i.e., the sum of the time spent on the workpiece, pick time, and walk time), the time per step, whether the step is completed within a specified pitch, etc.

[0063] The process monitor module 213 trains and applies process monitoring algorithms to monitor an assembly (or other physical) process. In some implementations, the process monitors the process to (1) determine which steps are being performed, (2) determine if all steps have been performed, (3) determine if steps are being performed out of sequence, (4) identify skipped steps, (5) identify steps that have been performed out of sequence, or (6) monitor the process for one or more combinations of these.

[0064] As described above, the image extraction module 209 performs extractions (e.g., applies pose estimation) on the video data, and the labeling module 211 applies labels. In some implementations, the process monitor module 213 receives video frames associated with one or more steps of a process (e.g., based on labeled start and end frames for the steps), receives video extraction data (e.g., pose estimation data and detected objects) corresponding to the one or more steps, and trains one or more process monitoring algorithms based on the two sets of data.

[0065] In some implementations, a process monitoring algorithm is trained for each step identified by the labeling. In some implementations, a single process monitoring algorithm is trained. In some implementations, a single process monitoring algorithm may not be used for the entire process (e.g., the entire assembly of a transmission from start to finish), but may be used for the entire process visible to a particular camera or set of cameras (e.g., the four steps of transmission assembly performed on an assembly line within the field of view of a particular camera device 152a).

[0066] The process monitoring algorithm that is trained may vary depending on the implementation. For example, the process monitoring algorithm may be an algorithm generated using supervised, semi-supervised, or unsupervised machine learning, depending on the implementation. Supervised machine learning methods include gradient boosted trees, random forests, support vector machines, neural networks, logistic regression (with regularization), linear regression (with regularization), stacking, naive Bayes, linear discriminant analysis, k-nearest neighbors, similarity testing, and / or other supervised learning models known in the art. Unsupervised machine learning methods include clustering (e.g., hierarchical, k-means, mixture models, DBSCAN, OPTICS algorithms, etc.), anomaly detection (e.g., local outlier factor), neural networks (e.g., autoencoders, deep belief nets, Hebbian learning, generative adversarial networks, self-organizing maps, etc.), and techniques for learning latent variable models (e.g., expectation-maximization algorithms, method of moments, blind signal separation, etc.), and / or other unsupervised learning models known in the art. Semi-supervised machine learning methods include generative models, low-density separation, graph-based methods, heuristic techniques, and / or other semi-supervised learning models known in the art.

[0067] For clarity and convenience, specific methods (e.g., the use of neural networks) are discussed herein, but it should be understood that these are examples and that other learning methods may be used and are within the scope of the present disclosure.

[0068] Before addressing specific example implementations of fixed and moving processes and statistically and deep learning-based algorithms, it may be helpful to describe the process monitor module 213 more abstractly. The process monitor module 213 receives extracted video information (e.g., pose data) generated from an incoming image stream (e.g., video) and generates an insight into which step is being performed and whether the end of the step is detected. For example, during deployment to a factory, a worker is videotaped driving a bolt. The video may be fed live to the image extraction module 209, which outputs the extracted video data. The process monitor module 213 obtains the extracted video data. Based on the extracted video data, the process monitor module 213 determines which step it believes is being performed at time 1 (e.g., driving bolt 1 as opposed to driving bolt 2). As video of the bolt being driven continues to be received and processed by the image extraction module 209, the process monitor module 213 uses this new extracted video data to update its view at time 2. The periodic updating of the view may vary based on the implementation (e.g., every frame, every 5 frames, every 10 frames, every X milliseconds, etc.). When the process monitor module 213 determines that there is a possible step completion, it makes a determination whether the step has taken place, and if so, updates its view with the completion of the step and the frame number of the step.

[0069] It should be appreciated that the systems and methods herein provide accurate real-time determination of which steps are being performed and whether the steps are complete. Thus, the systems and methods described herein provide real-time alerts (e.g., out-of-sequence or skipped steps) to workers so that corrective action may be taken if necessary. Thus, the systems and methods described herein may immediately improve processes by providing timely feedback to reduce errors, rather than simply post-mortem analysis or retrospective review of a worker's past performance.

[0070] The process monitoring module 213 described above may be used to monitor a fixed process (e.g., a process performed on a stationary work table), a moving process (e.g., a process performed on an intermittently or continuously moving assembly line or conveyor), or both. However, for clarity and convenience, a more detailed discussion is provided with reference to FIG. 4, which is an exemplary process monitoring module 213 shown according to one implementation. In the shown implementation, the process monitoring module 213 includes a fixed process module 402 and a moving process module 404.

[0071] The fixed process module 402 trains and applies process monitoring algorithms to monitor fixed assembly (or other physical) processes (e.g., assembly processes with a stationary worktable and / or assembly processes in which workpieces are stationary while steps are performed). As discussed above with reference to the process monitoring module 213, the process monitoring algorithms may be generated using different learning methods or combinations thereof, depending on the implementation. However, for clarity and convenience, two methods are described below: deep learning (using recurrent neural networks or transformers) and statistics-based methods. Thus, the fixed process module 402 in the illustrated implementation of FIG. 4 includes a deep learning module 412 and a statistics-based module 414.

[0072] In some implementations, the deep learning module 412 has the ability to learn models of objects in a supervised or unsupervised manner. In one such implementation, a model of an object has two main components: appearance and behavior. Appearance can be combined with rotation to allow for different viewpoints, distances, and types. In some implementations, behavior is divided into either different objects or different modes. To illustrate by example, a car going straight or turning left—both of which are two different modes of the same object, or two different objects.

[0073] In some implementations, the deep learning module 412 includes embedded code that defines or selects a model. A discriminator (not shown) within the deep learning module 412 selects / outputs an embedding code based on a time series of images. In some implementations, this provides a way to consider creating new models as well as updating existing models.

[0074] The deep learning module 412 generates correlation values ​​with other models / embedded codes, which allows it to learn to recognize the presence of recurring objects.

[0075] It should be recognized that unsupervised learning has the ability to learn appearance in a local way - visually similar objects are learned to be similar - whereas supervised learning uses information possessed by human labelers that may not be evident from the image alone to bring the embeddings closer together, thereby using behavior to bring the embeddings closer together.

[0076] Behavior can be learned by the deep learning module 412 by looking at changes in objects over time in the video stream. When similar-looking objects are clustered together (across frames), their trajectories can be learned by the deep learning module 412. A first-order approximation of behavior is linear motion, i.e., optical flow or velocity, in each distinct step. Depending on the similarity that the appearance embedding encodes, more complex trajectories may be learned (which allows for better tracking over time).

[0077] It should be further appreciated that although deep learning is discussed herein in relation to fixed processes (because deep learning module 412 is a submodule of fixed process module 402 in FIG. 4), the deep learning methods discussed herein may also be applied to mobile processes. However, for the sake of brevity, a separate description of deep learning for mobile processes is not provided below in relation to mobile process module 404.

[0078] In some implementations, the deep learning module 412 uses a recurrent neural network (RNN) (including varieties such as GRU and LSTM). The RNN is trained with examples of process steps. When fed sequential images, the network outputs which process has been completed. Depending on the implementation, the RNN may use pose data or other data above and beyond images as input. In some implementations, the deep learning module 412 uses a transformer. Similar to an RNN, a transformer-based model is fed a sequence of images or pose data and outputs the steps of the process. Both deep learning methods (i.e., RNN and Transformer) may encode a sequence of data into an embedding that is used by a decoder network to output the completion of the step of the process.

[0079] Such deep learning proposes "models" of the objects used by the person in the scene, and also a model of the person. These models allow system 100 to compare predictions (what should happen during a process step) with what happens to increase the confidence of the process step or to alert when the process is not being followed. Additionally, system 100 may indicate the confidence of step completion (similar to statistical process monitoring described below), generating an output of the steps that were completed, when those steps were completed, and the confidence that the steps were performed correctly.

[0080] The statistical-based module 414 trains and applies a statistical-based process monitoring algorithm to monitor the process. In some implementations, the statistical-based module 414 receives recorded examples with labeled endpoints.

[0081] The statistical-based module 414 may generate path points by tracking pose data based on active keypoints (e.g., the left wrist). The statistical-based module 414 combines this pose data into statistical averages of points, separated by some number of pixels, along the x and y dimensions with different probabilistic models such as Gaussian, Poisson, and Beta, among others. The statistical-based module 414 compares these statistical points with pose data from the testing process to determine the likelihood that someone is following the same path.

[0082] The statistics-based module 414 may generate endpoints, which are where the posture data ends when a step is accomplished. An example is screwing a bolt with a torque wrench -- the wrist pauses for a few seconds as the bolt is screwed in, giving an indication that the worker has paused. These endpoints are used to identify steps that are far apart within the image canvas.

[0083] The statistics-based module 414 may compare the similarity of the endpoints. Endpoints of different steps that are very close to each other may be grouped in such a way that their endpoints can be estimated based on other factors such as which step was completed previously, what tool is being used, and what time in the step's cycle.

[0084] The statistics-based module 414 may determine a frame length. Frame length is like dwell time at an endpoint—when someone screws in a bolt, they might spend 2 seconds at that endpoint. When they perform another task, such as clipping on a plastic part, they might spend 0.5 seconds. This is another signal that the statistics-based module 414 may use to help determine which step is being performed.

[0085] The statistics-based module 414 may determine the direction of entry or exit. The direction in which the endpoint is entered and the direction in which the endpoint is exited may indicate which step is being performed, as well as which step was performed previously and which step will be performed next.

[0086] The statistics-based module 414 may determine the order of steps. The order of steps may be a useful signal to identify which steps are being performed. However, the order of steps may also be a signal that may be negligible in some contexts (e.g., because the order is not very important). In some implementations, the order of steps performed is compared to the work instruction / training order, and workers are notified when they are not following the work standard. Typically, knowing if a worker has skipped a step entirely is more important than knowing if the step was performed out of sequence. Generally, it is not desirable to stop the line because a step is out of sequence.

[0087] The statistics-based module 414 may utilize object detection to determine whether a tool is present, such as a drill, wrench, screwdriver, etc. The presence of the tool may indicate what step is being performed.

[0088] In some implementations, the statistics-based module 414 combines and weights one or more of the above (path points, end points, end point similarity, frame length, incoming and / or outgoing direction, and step order) to periodically make a determination as to which steps are being performed and whether the steps have been completed. In some implementations, once a process is completed or time has elapsed, the information is sent to a gateway or cloud API for processing.

[0089] The weights of these pieces of information and how they are combined are process dependent and may be tuned. For example, a user setting up monitoring of a process may be presented with a user interface that allows the user to modify the combinations and / or weightings. The complexity of the user interface may vary from simple (e.g., a single slide bar) to complex (e.g., showing various weightings, speed thresholds for determining whether something is moving or stationary, distance thresholds for determining whether a keypoint is "close" enough to an end point, etc.). The variations may be based on implementation or user selection (e.g., selecting "advanced").

[0090] The movement process module 404 uses the movement of the process to help segment the steps. In some implementations, the labeling applied via the labeling module 211 includes bounding boxes. For example, in some implementations, the labeling module 211 labels bounding boxes for one or more of the step, the start frame, and the end frame. In some implementations, the labeling module 211 may also label key points on the object being assembled at two points (e.g., the beginning and the end) of the step.

[0091] Depending on the implementation, the move process module 404 may compensate for the movement of the workpiece in the move process in several ways. In some implementations, the move process module 404 compensates for the movement by normalizing the movement relative to the workpiece, i.e., the frame of reference may be the workpiece (e.g., not the camera's field of view) and the movement may be defined relative to the workpiece. It should be understood that such an implementation may be better at detecting (or not penalizing) the detection of steps performed further above or below the line than they did in the training instances.

[0092] In some implementations, the movement process module 404 uses keypoints within the bounding box (e.g., keypoints associated with a human wrist) to construct a 2D heatmap or Gaussian distribution that captures the movement because steps with low movement (e.g., drilling a hole) have fairly tightly packed points, while steps with high movement (e.g., torqueing a wrench) have scattered points.

[0093] In some implementations, the movement process module 404 determines a dominant vector of movement. For example, the movement process module 404 uses two labeled points associated with feature X in frame 10 and feature X in frame 200 and generates a dominant vector of movement based on those two points.

[0094] Using the dominant motion vector and frame length, the motion process module 404 may compare the last N pose points seen with the labeled examples received during training. This comparison may be performed at different intervals (e.g., every frame, every 5 frames, every 10 frames, every 20 frames, etc.) depending on the implementation.

[0095] Because the pieces being assembled during deployment are moving and may or may not be in the same location as the pieces assembled during training, and / or the camera may be bumped or shifted during deployment, the movement process module 404 applies normalization so that steps taken some distance (e.g., one foot forward or backward) from where the step was taken during training are not penalized and correctly identified. Referring to FIG. 5, in one such implementation, the movement process module 404 calculates a mean line perpendicular to the dominant vector of movement and then normalizes the positions of the points of the N poses relative to the dominant vector of movement 508 by shifting the mean line of test points perpendicular to the dominant vector of movement (i.e., test points extracted from video taken while testing the algorithm or during camera deployment) to the mean line of labeled examples perpendicular to the dominant vector of movement (i.e., labeled examples received and labeled during training). For example, referring to FIG. 5, point 502a is shifted to the position of point 502b.

[0096] The movement process module 404 performs a similarity score between the steps currently being performed and captured in the video and those steps captured during training. The similarity score may vary (e.g., depending on the implementation, use case, user-selected criteria, etc.). Examples of similarity scores include forearm angle similarity (the vector between the elbow keypoint and the wrist keypoint), distance from 0 shifted (e.g., steps that appear closer to where they would be expected to be based on training examples of the steps are more likely to be the same step), etc.

[0097] In some implementations, the moving process module 404 applies criteria to determine which steps are being performed and whether the steps are complete. In some implementations, the criteria include, for example, a threshold based on the total score of a set of points. In some implementations, the moving process module 404 pairs the closest points (such as until no points remain in the test set (generated from the step being performed) or the labeled set (generated from the execution of step X during training). For example, the moving process module pairs point 502b with point 504. In some implementations, the moving process module 404 then calculates the distance between the paired points, e.g., points 502b and 504, as indicated by line 506. The distance calculations for each point pair are summed by the moving process module 404, and when the summed value (i.e., similarity score) is less than a threshold, the moving process module 404 determines that step X is being performed and complete. Note that while a "distance" calculation is described, in some implementations, the system may use a displacement calculation.

[0098] In some implementations, when the move process monitor 404 or the process monitor module 213 determines that a step is complete, the labeling module 211 applies a completed step flag. The completed step flag may be associated with a frame number and used to calculate step time and total cycle time. These times may be used by one or more components of the system. For example, in some implementations, the labeling module 211 may use the time to apply a “good” label, and the process optimization module 219 may use this label to generate suggested optimizations. In another example, in some implementations, the optimization module 219 may analyze instances that take less than a threshold amount of time to generate suggested optimizations without having to label the instance as “good.”

[0099] In some implementations, the process monitor 213 or its subcomponents may generate and transmit alerts. For example, the process monitor module 213 may generate one or more of a “skipped step” alert or an “out-of-sequence step” alert to an operator, e.g., via an electronic display at the operator's work station. As another example, the process monitor module 213 may generate an alert with a label, e.g., “good,” to provide immediate feedback to the operator. In some implementations, the alert may not be via a display or may be for display only. For example, a red light may be flashed or the line may be paused or stopped as part of a “skipped critical step” alert. In some implementations, the alerts generated and / or transmitted may be customizable. For example, it may be desirable to alert an operator only to a “critical” step that was skipped, while elsewhere in the process (or in a different plant) it may be desirable to alert an operator to all skipped steps. As another example, during a worker's training period, or for the first few cycles in which the worker interacts with a new workpiece or performs a new / unfamiliar step, it may be desirable to provide "good" and "bad" labels to help the worker learn, but phase this out after a predefined number of cycles so that the worker is focused on the working workpiece and not the display. It should be recognized that the warnings and customizations herein are merely examples and that other warnings and customizations exist and are within the scope of this specification.

[0100] Currently, existing systems rely on regular security cameras installed throughout a facility. However, because a lot of bandwidth and equipment is required to handle the many camera feeds and large amounts of central storage are needed to store the footage, such systems are expensive to deploy and use the cameras sparingly, thus having only partial visibility of the operations. Today, when a problem occurs, plant operators go to each camera's footage and try to match the video to different instances (e.g., different cars) so that the operators can determine which cars were affected and the cause. This typically involves spending hours combing through video footage to identify the cause.

[0101] The tracking module 217 provides a visibility and traceability system for the manufacturing facility. In some implementations, the tracking module 217 provides a search engine for the manufacturing facility that integrates with the manufacturer's existing manufacturing execution system (MES) to become a video system of record for all products manufactured within the facility.

[0102] The search engine provided by the tracking module 217 may vary depending on the implementation. In some implementations, the search engine receives one or more of a period of time and an identifier. Examples of identifiers include, but are not limited to, one or more of a unit identifier (e.g., chassis number, serial number, VIN number, barcode, QR code, etc.), a product identifier (e.g., brand, model, and trim), a worker (e.g., employee or robot ID), a work station identifier (e.g., workbench a), etc. Examples of time periods include one or more of recent time periods (e.g., last hour, last shift, last day, last 3 days, last 5 days, last week, last 2 weeks, last month, last 3 months, last quarter, year to date, etc.) and custom time periods (e.g., presenting a calendar and / or clock and allowing a user to query a specific time period, such as from 3:00 PM on Tuesday, September 15, 2020 to noon on Thursday, September 24, 2020).

[0103] In some implementations, in response to a query, a search engine presents a set of query results. The presentation of the query results may vary based on the implementation and one or more of the options selected by the user. For example, a search for results for a given product type (e.g., 2WD transmissions) over a period of time may be presented as a list of video clips from camera devices at work stations that worked on that product type (i.e., 2WD transmissions in this example). The results may be sorted chronologically based on timestamps from the video, grouped by product ID (e.g., so that all clips related to 2WD transmission A are presented together), grouped by worker or work station, etc.

[0104] In some implementations, metrics are presented along with or as part of the query result set. For example, assuming a search was for 2WD transmissions over the last 24 hours, in some implementations, the tracking module 217 may present for display metrics related to the average step time, average cycle time, and number of omissions (e.g., skipped steps, line stoppages, etc.) over the last 24 hours. As another example, when a video clip associated with the result set is selected so that the user can view the recorded video, additional metrics may be presented (e.g., the time it took the workers in the recording to perform the recorded step or cycle, statistics related to the recorded workers completing the step or cycle (e.g., box plots showing the distribution of workers' performance for the step or cycle, box plots showing the distribution of all workers' performance, etc.), a label (e.g., "good"), a detected omission (e.g., "failed to drill bolt 3"), a suggestion (e.g., move the drill to the left side of the table and have the worker drill with their left hand), etc.).

[0105] In some implementations, the tracking module 217 integrates with one or more of the manufacturer's data acquisition and storage systems. Examples of data acquisition and storage systems include, by way of example and not limitation, programmable logic controller (PLC) controllers, manufacturing execution systems (MES), enterprise resource planning (ERP) systems, supervisory control and data acquisition (SCADA) systems, etc. Through integration, the tracking module 217 captures data for each assembly work station on the floor. Examples of captured data include, by way of example and not limitation, the type of product being built (e.g., hybrid, 2WD, all-wheel drive, etc.), the start time of the product's appearance at the work station, additional product-specific data required by the customer (such as the type of vehicle for a related product being built), etc.

[0106] The tracking module 217 is integrated with the camera devices 152a-n. In some implementations, the tracking module is server-based. For example, the tracking module 217 is included in the server 162 (not shown) and integrated with the camera devices 152a-n via the network 150. In some implementations, the tracking module 217 is included in the camera devices 152a-n, which communicate and cooperate to provide the features described herein. In some implementations, the tracking module 217 includes subcomponents distributed between the server 162 (not shown) and the camera devices 152a-n, which cooperate to provide the features and functionality described herein.

[0107] Camera devices 152a-n include built-in video storage (e.g., memory 206 and / or database 208) and processing (e.g., processor 204) to document the work being done on any product and automatically index the data for stakeholders (e.g., plant managers, supervisors, process engineers, etc.). In some implementations, the built-in video storage is 1 TB. In some implementations, the built-in storage is sufficient to store 30 days' worth of high-definition video and an additional 30 days' worth of low-resolution video. In some implementations, older video data is archived. For example, after 30 days, the high-resolution video is downsampled to low-resolution video, and after another 30 days, the low-resolution video is archived from camera device 152 (e.g., on a network-accessible storage device). In some implementations, video is overwritten when the built-in video storage is filled. For example, assuming 30 days' worth of video is stored in the video storage, on the 31st day, video begins to overwrite the video stored on the first day (i.e., on a first-in, first-out basis).

[0108] In some implementations, using a web dashboard provided by the tracking module 217, an interested party can enter a product's unique ID (e.g., body number) into a search bar, and the tracking module 217 will receive the query and provide relevant video data that is automatically fetched for the appropriate time from each camera deployed in the facility so that the production of that product may be viewed from start to finish.

[0109] In some implementations, the tracking module 217 is not limited to queries based solely on unique identifiers associated with workpieces. For example, depending on the implementation, the tracking module 217 may enable queries related to one or more of a particular workstation, a particular worker, a particular product or unit identifier, a particular posture or movement (e.g., to identify where a worker is required to reach above or behind their head or lift and twist, or how many times the worker performs such actions, etc.). In one implementation, the tracking module 217 may identify a reach based on posture data, e.g., based on the distance between a wrist keypoint and a shoulder keypoint, which in some implementations may be compared against the sum of the distance from the shoulder keypoint to the elbow keypoint and the distance from the elbow keypoint to the wrist keypoint. In one implementation, the tracking module 217 may identify a bend, head lift, or stoop based on posture data, e.g., based on the distance between a hip keypoint and a shoulder keypoint.

[0110] Given the built-in video storage and processing of camera devices 152a-n, manufacturers may deploy hundreds or even thousands of cameras, given the relatively low equipment and IT costs / burden, which provides greater traceability and visibility into products built at the manufacturer's facilities.

[0111] The process optimization module 219 analyzes data captured by the camera device 152 or generated by the assembly monitoring module 203 or its subcomponents to generate insights.

[0112] For example, using motion data describing a person's movements, how long they spend on each step, the total cycle time, the "wasted" time between cycles, pick time, walk time, and where on the pitch the steps begin and end, the process optimization module 219 may determine whether a person is underworked or overworked and where they are prone to making errors. For example, when the beginning of a pitch in a movement process is associated with "0" and the end of a pitch is associated with "100," when a worker consistently starts (e.g., at -12) and / or finishes early (e.g., at 70), it may indicate that the worker is underworked (e.g., the task is not good enough / the line is moving too slowly in that section, etc.) or is very efficient; when a worker consistently completes a task outside of the pitch (e.g., at 108), it may indicate a problem, for example, the worker is overworked, inefficient, the process is more complex and time-consuming, the worker consistently needs to correct or address an upstream issue, etc. The process optimization module 219 may automatically generate insights, for example, that the operator at station 3 is constantly overworked and the operator at station 2 has 45 seconds of downtime between every cycle, which is a lot of wasted time, so it would be beneficial to move two steps from station 3 to station 2.

[0113] The labeled examples used for training are ground truth, and in some implementations, a human may input other ground truth values, such as how long a human should be "free" on average between cycles, that the process optimization module 219 may evaluate against and average to see where optimizations are possible.

[0114] The process optimization module 219 may provide insights in addition to or instead of improvements. For example, in some implementations, the process optimization module 219 may surface best practices, which may identify that an operator may have found a better (but non-standard) way and therefore this way should be adopted by others. Alternatively, the process optimization module 219 may surface an operator performing a task unsafely, which may result in the behavior being stopped, either of which may be surfaced by the same mechanisms described above. Using object orientation and / or user posture, the process optimization module may discern whether a tool is being used correctly and ergonomically and suggest changes.

[0115] Other examples of the types of insights that the process optimization module 219 may surface include: (1) identifying whether it is more optimal for a particular process or work station to use the left hand rather than reaching farther with the right hand; (2) identifying whether steps can be moved around between successive work stations on an assembly line for greater efficiency; and (3) moving steps to balance the workload (e.g., if station 1 has five steps and can be completed very quickly, while station 2 has 12 steps and takes a very long time—an insight may be generated recommending moving a step from station 2 to station 1 to balance the workload).

[0116] Exemplary Methods 6 is a flow diagram of an example method for generating a process monitoring algorithm, according to one implementation. At block 602, videos of example instances are received for training by the image extraction module 209 and the labeling module 211. In some implementations, each video represents an example of a workpiece undergoing one or more (common) steps associated with the process being monitored.

[0117] Although blocks 604 and 606 are shown in parallel, they may be performed serially in some implementations. In block 604, the image extraction module 209 performs image extraction. For example, the image extraction module 209 extracts pose data and / or detects objects. In block 606, the labeling module 211 applies labels to the videos of the example instances. In some implementations, at least one set of labels (e.g., a set identifier, a start frame, an end frame, and any other labels used in a particular implementation, which may include one or more regions of interest, one or more keypoints, etc.) is received based on input from a human user labeling the videos of the example instances. In some implementations, other sets of labels may be received based on user input, i.e., by a user labeling two or more videos, or may be automatically generated by the labeling module 211 based on the user-input labels. In block 608, the process monitoring module 213 trains a process monitoring algorithm using the image extraction data generated in block 604 and the labels applied in block 606.

[0118] 7 is a flow diagram of an exemplary method for monitoring a process according to one implementation. At block 702, video of the process is received by the image extraction module 209 and the process monitoring module 213. For example, live video of a new instance of a workpiece undergoing a process being captured by the camera device 152 is received by the image extraction module 209. At block 704, the image extraction module 209 performs image extraction on the video received at block 702. For example, the image extraction module 209 periodically generates pose data and detects objects, for example, every x frames or y milliseconds. At block 706, the process monitor module 213 applies a process monitoring algorithm, generating step data at block 708. For example, the process monitor module 213 generates step data such as an identification of which step the process monitor module 213 believes is being performed, whether the step is complete, and, depending on the implementation, may include one or more associated labels, warnings, or suggestions (e.g., a skipped step warning, a step performed out of sequence warning, an improper posture warning, "stand up straight," "ideal," etc.).

[0119] 8 is a flow diagram showing another exemplary method 800 of monitoring a process, according to one implementation. At block 802a, the image extraction module 209 provides extracted information from the video at time 0 to the process monitor 213. At block 804a, the process monitor 213 receives the extracted information at time 0 and makes a decision. At block 802b, the image extraction module 209 provides extracted information from the video at time 1 to the process monitor 213. At block 804b, the process monitor 213 receives the extracted information at time 2 and updates the decision made by the process monitor at block 804a. As video continues to be received, blocks 802 and 804 are repeated, as indicated by the ellipsis between blocks 802b and 804b and blocks 802n and 804n.

[0120] In some implementations and use cases, multiple workpieces and / or work stations may be visible within the field of view of the camera device. In some implementations, the process monitor 213 determines the object at block 806n. For example, the process monitor determines one or more of which of multiple visible workpieces is being worked on or its associated state should be updated, and which tool, if any, is present. At block 808n, the process monitor 213 makes a determination that may be output as step data. For example, the process monitor updates its view of which step is being performed and whether the step is complete (e.g., the state of the object). As indicated by the arrow from block 808n to 806n, the view may be returned to block 804n so that it can be subsequently used and / or updated at time n+1 (not shown). Although not shown, it should be understood that blocks similar to blocks 806n and 808n may exist for time 0 and time 1.

[0121] Other considerations In the above description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it should be understood that the technologies described herein may be practiced without these specific details. Furthermore, various systems, devices, and structures are shown in block diagram form to avoid obscuring the description. For example, various implementations are described as having particular hardware, software, and user interfaces. However, the present disclosure applies to any type of computing device capable of receiving data and commands, as well as any peripheral device that provides services.

[0122] In some cases, various implementations are presented herein in terms of algorithms and symbolic representations of operations on data bits within a computer memory. An algorithm is here, and generally, conceived to be a self-consistent set of operations leading to a desired result. The operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0123] For ease of description, some elements of the systems and / or methods may be referred to using labels such as first, second, third, etc. These labels are intended to aid in distinguishing between the elements and do not necessarily imply any particular order or hierarchy unless specifically indicated.

[0124] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As will become apparent from the discussion that follows, unless otherwise noted, throughout this disclosure, discussions utilizing terms including "processing," "calculating," "computing," "determining," "displaying," and the like will be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the computer system's registers and memory into other data that is similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display device.

[0125] Various implementations described herein may relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium, including, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magnetic disks; read-only memory (ROM); random-access memory (RAM); EPROM; EEPROM; magnetic or optical cards; flash memory, including USB keys with non-volatile memory; or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0126] The technology described herein may take the form of an all-hardware implementation, an all-software implementation, or an implementation containing both hardware and software elements. For example, the technology may be implemented in software, including but not limited to firmware, resident software, microcode, etc. Furthermore, the technology may take the form of a computer program object accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable medium can be any non-transitory storage device that can contain, store, convey, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device.

[0127] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements may include local memory used during the actual execution of the program code, mass storage, and cache memory for temporary storage of at least some program code to reduce the number of times the code must be retrieved from mass storage during execution. Input or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0128] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems, storage devices, remote printers, and so on through intervening private and / or public networks. Wireless (e.g., Wi-Fi™) transceivers, Ethernet adapters, and modems are just a few examples of network adapters. Private and public networks may have any number of configurations and / or topologies. Data may be transmitted between these devices over the networks using a variety of different communication protocols including, for example, various Internet layer, transport layer, or application layer protocols. Data may be transmitted over a network using, for example, Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Hypertext Transfer Protocol (HTTP), Secure Hypertext Transfer Protocol (HTTPS), dynamic adaptive streaming over HTTP (DASH), Real-time Streaming Protocol (RTSP), Real-time Transport Protocol (RTP) and real-time transport control protocol (RTCP), Voice over Internet Protocol (VOIP), File Transfer Protocol (FTP), WebSocket (WS), Wireless Access Protocol (WAP), various messaging protocols (SMS, MMS, XMS, IMAP, SMTP, POP, WebDAV, etc.), or other known protocols.

[0129] Finally, the structure, algorithms, and / or interfaces presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method blocks. The required structure for a variety of these systems will be apparent from the above description. Moreover, this specification is not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings herein, as described herein.

[0130] The foregoing description has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the specification to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. As will be understood by those skilled in the art, the specification may be embodied in other specific forms without departing from the spirit or essential features thereof. Likewise, the specific naming and division of modules, routines, features, attributes, methods, and other aspects is not necessary or critical, and mechanisms for implementing the specification or its features may have different names, divisions, and / or forms.

[0131] Furthermore, the modules, routines, features, attributes, methods, and other aspects of the present disclosure may be implemented as software, hardware, firmware, or any combination thereof. Also, whenever components herein, of which modules are examples, are implemented as software, the components may be implemented as a standalone program, as part of a larger program, as multiple separate programs, as a statically or dynamically linked library, as a kernel-loadable module, as a device driver, and / or in any and all other manners now known or later known. Furthermore, the present disclosure is in no way limited to implementations in a specific programming language or for a specific operating system or environment. [Explanation of symbols]

[0132] 100 systems 150 Cloud, Network 152a~n Camera Device 156 client devices 162 servers 202 Communication Unit 203 Assembly Monitoring Module 204 processors 206 memory 208 databases 209 Image Extraction Module 210 Communication Bus 211 Labeling Module 212 Input Devices 213 Process Monitoring Module, Process Monitor Module, Process Training Monitor, Process Monitor 214 Output Devices 217 Tracking Module 219 Process Optimization Module 402 Fixed Process Module 404 Migration Process Module 412 Deep Learning Module 414 Statistics-based Modules 502a points 502b points 504 points 508 Dominant Movement Vector 800 ways

Claims

1. receiving, using one or more processors, image data, the image data including a first training video depicting performance of one or more steps on a first workpiece; applying, using the one or more processors, a first set of labels to the first training video based on user input, the set of labels including a start frame label associated with a frame where a step begins, an end frame label associated with a frame where the step ends, and a step identifier that identifies the step; using the one or more processors, performing an extraction of information related to the first training video on the image data, thereby generating extracted information, the extracted information including first extracted image information related to the first training video; and using the one or more processors, training an assembly process monitoring algorithm based on the extracted information and the first set of labels.

2. applying the first set of labels to the first training video based on user input; receiving from a user a plurality of sets of bounding boxes, each set of bounding boxes associated with a single workpiece, the first set of labels including the first set of bounding boxes; training an object detection algorithm based on the set of bounding boxes; the extracting includes applying the object detection algorithm, and the extracted image information includes object detection. The method of claim 1.

3. 3. The method of claim 2, wherein the first set of bounding boxes includes a first bounding box around the first workpiece in a start frame, a second bounding box around the first workpiece in an intermediate frame, and a third bounding box around the first workpiece in an end frame, and training the object detection algorithm includes applying a Deal Learning object detection network.

4. The method of claim 2 , wherein the first set of bounding boxes represents a first workpiece at the beginning, intermediate stages, and end of a cycle within a field of view of a camera device.

5. applying the first set of labels to the first training video based on user input; receiving a set of polygons from a user, the set of polygons comprising one or more of a pick area polygon, a walking area polygon, and a pitch polygon; the first set of labels includes the set of polygons; The method of claim 1.

6. The method of claim 1 , wherein performing extraction on the image data generates one or more of pose data and object detection.

7. The method of claim 1, further comprising: automatically applying a second set of labels to second training video image data based on the first set of labels, wherein the first set of labels is applied based on user input; performing extraction on second training video image data, thereby generating second extracted information, wherein the training video image data includes the second training video image data; The method of claim 1 , wherein the training of the assembly process monitoring algorithm is further based on the second extracted information and the second set of labels.

8. receiving live video image data after training the assembly process monitoring algorithm; performing extraction on the live video image data, thereby generating extracted information of the live video; applying the assembly process monitoring algorithm to the live video using the extracted information of the live video; The method of claim 1 further comprising the step of: presenting step data.

9. performing extraction on the live video image data includes obtaining pose data and object detection, and applying the assembly process monitoring algorithm and presenting the step data includes: applying the assembly process monitoring algorithm to extracted data of a first live video associated with a first time to generate a first view; presenting first step data including the first opinion, the first opinion including a step identifier and whether the step is complete; generating a second view by applying the assembly process monitoring algorithm to extracted data of a second live video associated with a second time and updating the first view; and presenting second step data including the second opinion.

10. 1. A system comprising: one or more processors; a memory storing instructions that, when executed by the one or more processors, receiving image data, the image data including a first training video depicting performance of one or more steps on a first workpiece; applying a first set of labels to the first training video based on user input, the set of labels including a start frame label associated with a frame at which a step begins, an end frame label associated with a frame at which the step ends, and a step identifier identifying the step; performing an extraction of information related to the first training video on the image data, thereby generating extracted information, the extracted information including first extracted image information related to the first training video; A system including a memory that causes training an assembly process monitoring algorithm based on the extracted information and the first set of labels.

11. applying the first set of labels to the first training video based on user input when performed by the one or more processors. receiving from a user a plurality of sets of bounding boxes, each set of bounding boxes associated with a single workpiece, the first set of labels including the first set of bounding boxes; training an object detection algorithm based on the set of bounding boxes; the extracting includes applying the object detection algorithm, and the extracted image information includes object detection. The system of claim 10.

12. 12. The system of claim 11, wherein the first set of bounding boxes includes a first bounding box around the first workpiece in a start frame, a second bounding box around the first workpiece in an intermediate frame, and a third bounding box around the first workpiece in an end frame, and training the object detection algorithm includes applying a Deal Learning object detection network.

13. The system of claim 11 , wherein the first set of bounding boxes represent a first workpiece at the beginning, intermediate stages, and end of a cycle within a field of view of a camera device.

14. applying the first set of labels to the first training video based on user input when performed by the one or more processors. receiving from a user a set of polygons, the set of polygons comprising one or more of a pick area polygon, a walking area polygon, and a pitch polygon; the first set of labels includes the set of polygons; The system of claim 10.

15. The system of claim 10 , wherein performing extraction on the image data generates one or more of pose data and object detection.

16. The system according to claim 15, wherein when executed by said one or more processors, automatically applying a second set of labels to second training video image data based on the first set of labels, wherein the first set of labels is applied based on user input; performing extraction on second training video image data, thereby generating second extracted information, the training video image data including the second training video image data; 11. The system of claim 10, wherein the training of the assembly process monitoring algorithm is further based on the second extracted information and the second set of labels.

17. When executed by the one or more processors, receiving live video image data after training the assembly process monitoring algorithm; performing extraction on the live video image data, thereby generating extracted information of the live video; applying the assembly process monitoring algorithm to the live video using the extracted information of the live video; and and presenting step data.

18. performing extraction on the live video image data includes obtaining pose data and object detection, and applying the assembly process monitoring algorithm and presenting the step data; applying the assembly process monitoring algorithm to extracted data of a first live video associated with a first time to generate a first view; presenting first step data including the first opinion, the first opinion including a step identifier and whether the step is complete; generating a second view by applying the assembly process monitoring algorithm to extracted data of a second live video associated with a second time and updating the first view; and presenting second step data including the second opinion.

Images