Method and system for automated repetitive step and cycle detection of manual assembly line operations

Abstract

A system for determining the quantitative accuracy of a test movement relative to a reference movement includes a display output device, a memory, and a processor operably connected to the display output device and the memory. The memory stores motion capture data and programming instructions. The processor executes the programming instructions to determine the quantitative accuracy of the test movement relative to the reference movement. A method for determining quantitative accuracy that can be performed by the processor includes receiving, with the processor, motion capture data including a reference movement and a test movement. The motion data is split into individual movements, and the test movement is aligned with the reference movement. The processor calculates the quantitative accuracy of the test movement relative to the reference movement, and generates a visualization representing the test movement using the display output device. The calculated accuracy is encoded into the visualization.

Description

Technical Field

[0001] The present disclosure relates generally to human motion sensing, and more particularly to automatic analysis of human motion for a defined process. Background Art

[0002] Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0003] Aspects of human movement can have a significant impact on, for example, the way tools are designed, the way workspaces are laid out, or the way tasks are performed. Understanding how the human body can move and interact with objects and the environment can lead to more ergonomic tools, more efficiently navigated workspaces, and more intuitively performed tasks. However, the range of possible human movements and postures is wide, and simple tasks such as lifting a cup, pointing a direction, or turning a screw are often caused by a complex collection of biomechanical interactions. This relationship of simple results caused by complex movements can make human movement and posture extremely difficult to quantify or understand in a meaningful or practical way.

[0004] As used herein, "movement" includes one or more of "motion" (i.e., relative change in position of a body part) and "posture" (i.e., relative change in orientation of a body part). Due to the complex nature of biomechanical interactions underlying even simple actions, human movement is often evaluated based on results rather than analysis of the movement itself. For example, it is often much easier to evaluate a ball thrown by a pitcher, a painting drawn by an artist, etc. than to evaluate the movements performed to achieve those results. However, considering the results may have limited utility in training others to accurately perform potential movements.

[0005] For example, manual and even semi-automatic assembly lines rely on workers to accurately perform complex movement sequences in order to maintain assembly quality and productivity levels. Specifically, the actions in the sequence must be performed in the correct order, within a defined time period, and with the correct motions and postures. While considering whether the results of a sequence are accurate can be done simply by considering whether the assembly was completed without defects within a defined time period, calculating whether the motions and postures are optimal, or, for example, identifying when and how the assembly deviates from the prescribed sequence, can be much more difficult to determine.

[0006] Technologies have been developed to help automatically identify human movement. For example, motion capture technology and computer vision have been used to generate three-dimensional models that can be used to identify posture or orientation, and detect changes in posture or orientation to identify motion or gesture. However, simply identifying specific motion or gesture has limited utility for quantitative calculation accuracy.

[0007] In an example, a smart assembly workplace provides computer-supported instructions for training workers to perform an assembly sequence. Prior to training, an experienced worker performs the movements required for the assembly sequence and inputs data, which includes images of each step of the assembly. The experienced worker then identifies the assembly steps associated with the data to define the actions to be taken and the position of the worker's hands during each step, as well as the target time to complete the step. Inexperienced workers using the workplace can be guided through the assembly sequence based on the actions identified for each step, can be notified if their hands are in the wrong orientation during a step, and can be evaluated by comparing their execution time to the target time for the sequence.

[0008] However, recording the position of the experienced worker's hand at each step is an observation of the result of the movement or gesture, not the movement or gesture itself. In other words, identifying that the inexperienced worker's hand is in an incorrect position does not reveal when or how the worker's hand deviates from the movement performed to reach that position, or provide guidance on correct movement. The identification of an inaccurate position indicates an inaccurate movement or gesture, but does not quantitatively assess when and how the movement or gesture deviates, and therefore has limited utility in correcting the deviated movement.

[0009] Therefore, a system that can quantify the accuracy of human movement would be beneficial. A system that does not require extensive training data collection and is usable when reference data is extremely limited, or when movement is difficult to describe or perform would also be beneficial. A system that provides guidance for correcting deviated movement would also be beneficial. A system that enables comparison of movement to multiple steps, and a system that can correlate step information in reference data with test data would also be beneficial. Summary of the invention

[0010] In order to determine how a test movement deviates from a reference movement, how to correct the test movement, and / or how the deviation affects the movement result, a method for calculating the quantitative accuracy of a test movement relative to a reference movement includes: receiving motion capture data including a reference movement and a test movement with a processor. The test movement is aligned with the reference movement via the processor. The processor calculates the quantitative accuracy of the test movement relative to the reference movement, and generates a visualization representing the test movement with a display output device. The calculated accuracy of the test movement relative to the reference movement is encoded into the visualization.

[0011] In some embodiments, the visualization representing the test movement comprises a timeline. The calculated accuracy of the test movement relative to the reference movement is encoded into the timeline to indicate quantitative values ​​of the accuracy of the reference movement at different time points along the timeline.

[0012] In some embodiments, the reference movement is represented as a first sequence of frames, each including motion capture data corresponding to a respective time interval of the first sequence, and the test movement is represented as a second sequence of frames, each including motion capture data corresponding to a respective time interval of the second sequence. Aligning the test movement with the reference movement includes mapping each frame of the second sequence for the test movement to at least one frame of the first sequence for the reference movement such that a sequential order of the mapped frames in each of the reference movement and the test movement relative to each other is preserved.

[0013] In some embodiments, the second sequence of frames for the test movement is mapped to the first sequence of frames for the reference movement via a dynamic time warp alignment process that accounts for a desynchronization between the test movement and the reference movement.

[0014] In some embodiments, the method further comprises: generating, with a processor and a display output device, a visualization representing the reference movement, the visualization comprising a first timeline formed by the first sequence of frames. The visualization representing the test movement comprises a second timeline formed by the second sequence of frames. The calculated accuracy of the test movement relative to the reference movement is encoded into the timeline to indicate quantitative values ​​of the accuracy of the reference movement at different time points along the timeline.

[0015] In some embodiments, the processor is further configured to warp the second timeline with reference to the mapping generated by the dynamic time warping process. The warping includes thickening and compressing portions of the second timeline representing portions of the test movement that are performed slower relative to corresponding portions of the reference movement. The quantitative amount of the thickening and compression is based on the relative rate between the corresponding portions. The warping additionally includes thinning and stretching portions of the second timeline representing portions of the test movement that are performed faster relative to corresponding portions of the reference movement. The quantitative amount of the thinning and stretching is based on the relative rate between the corresponding portions.

[0016] In some embodiments, the motion capture data includes: a record describing at least one of a motion, a posture, an orientation, and a position performed over time for each of the test move and the reference move. The quantitative accuracy of the test move relative to the reference move is calculated with reference to a quantitative difference over time in at least one of the motion, posture, orientation, and position between the test move and the reference move.

[0017] In some embodiments, the motion capture data includes a plurality of individual test moves including the test move.

[0018] In some embodiments, the plurality of individual test moves are in the form of a continuous multi-move cycle.The processor is further configured to split the continuous multi-move cycle into the individual test moves when aligning the test move with the reference move.

[0019] In some embodiments, the processor is further configured to calculate a respective quantitative accuracy of each individual test movement relative to a reference movement. The processor is further configured to generate a visualization representing each individual test movement using a display output device, wherein the calculated accuracy of the test movement relative to the reference movement is encoded into the visualization.

[0020] A system for determining the quantitative accuracy of a test move relative to a reference move comprises: a display output device, a memory, and a processor. The memory is configured to store programming instructions and motion capture data, the motion capture data comprising a reference move and a test move. The processor is operably connected to the display output device and the memory. The processor is configured to execute the programming instructions to determine the quantitative accuracy of the test move relative to the reference move. The determination of the accuracy comprises: aligning the test move with the reference move, calculating the quantitative accuracy of the test move relative to the reference move, and generating a visualization representing the test move using the display output device, wherein the calculated accuracy of the test move relative to the reference move is encoded into the visualization.

[0021] In some embodiments, the reference movement is represented as a first sequence of frames, each of which includes motion capture data corresponding to a corresponding time interval of the first sequence. The test movement is represented as a second sequence of frames, each of which includes motion capture data corresponding to a corresponding time interval of the second sequence. Aligning the test movement with the reference movement includes: mapping each frame of the second sequence for the test movement to at least one frame of the first sequence for the reference movement, such that the sequence order of the mapped frames in each of the reference movement and the test movement relative to each other is preserved.

[0022] In some embodiments, the processor is further configured to map the second sequence of frames for the test movement to the first sequence of frames for the reference movement via a dynamic time warp alignment process that accounts for desynchronization between the test movement and the reference movement.

[0023] In some embodiments, the determination of accuracy further comprises: generating, using a display output device, a visualization representing a reference movement, the visualization comprising a first timeline formed by frames of a first sequence. The visualization representing the test movement comprises a second timeline formed by frames of a second sequence. The calculated accuracy of the test movement relative to the reference movement is encoded into the timeline so as to indicate quantitative values ​​of the accuracy of the reference movement at different time points along the timeline. The processor is further configured to warp the second timeline with reference to a mapping generated by a dynamic time warping process. The warping comprises: thickening and compressing a portion of the second timeline representing a portion of the test movement that is performed slower relative to a corresponding portion of the reference movement. The quantitative amount of the thickening and compression is based on a relative rate between the corresponding portions. The warping further comprises: thinning and stretching a portion of the second timeline representing a portion of the test movement that is performed faster relative to a corresponding portion of the reference movement. The quantitative amount of the thinning and stretching is based on a relative rate between the corresponding portions.

[0024] In some embodiments, the system further comprises: a motion capture device configured to transfer the motion capture data to the memory.

[0025] In some embodiments, the motion capture device is a pair of motion capture gloves.

[0026] This summary is intended only to introduce subject matter related to quantitatively calculating the accuracy of a test move relative to a reference move, which is discussed in more detail in the detailed description, drawings, and claims, and is not intended to limit the scope of the present disclosure in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The foregoing aspects and other features of the present disclosure are explained in the following description considered in conjunction with the accompanying drawings.

[0028] Figure 1 is a schematic diagram of an exemplary embodiment of a system for calculating accuracy of continuous movement data according to the present disclosure.

[0029] Figure 2 It depicts Figure 1 A functional block diagram of an exemplary embodiment of the functional components of a system.

[0030] Figure 3 is a flow chart depicting an exemplary embodiment of a method for quantitatively calculating the accuracy of continuous movement data.

[0031] Figure 4 is used to receive mobile data Figure 1 An image of an exemplary embodiment of a graphical user interface ("GUI") of the system.

[0032] Figure 5 is a graph used to compare movement data. Figure 1 An image of an exemplary embodiment of a GUI for the system.

[0033] Figure 6a is a graph used to compare movement data. Figure 1 An image of another exemplary embodiment of a GUI for the system.

[0034] Figure 6b is a graph used to compare movement data. Figure 1 An image of another exemplary embodiment of a GUI for the system.

[0035] Figure 7 is a visual depiction of an exemplary embodiment of a process of aligning motion data from different motions using dynamic time warping.

[0036] Figure 8 is a graph used to compare movement data. Figure 1 An image of another exemplary embodiment of a GUI for the system.

[0037] Figure 9a It depicts the splitting of mobile data. Figure 1 An image of an exemplary embodiment of a GUI for the system.

[0038] Figure 9b It depicts the mobile data after it is split. Figure 9a An image of another exemplary embodiment of a GUI.

[0039] Fig.10 is a graph of move time, mean move error, and split confidence for motion capture data where multiple consecutive moves occur over time.

[0040] Fig.11 is a graph of movement timing of motion capture data, where multiple sequential movements occur over time.

[0041] Fig.12 is a graph used to compare movement data. Figure 1 An image of another exemplary embodiment of a GUI for the system.

[0042] Fig.13 is a visual depiction of another exemplary embodiment of a process of using dynamic time warping to align semantic information in motion data from different motions.

[0043] Fig.14 is a graph used to compare movement data. Figure 1 An image of another exemplary embodiment of a GUI for the system.

[0044] Fig.15 is a graph used to compare movement data. Figure 1 An image of another exemplary embodiment of a GUI for the system.

[0045] Fig.16 is a graph used to compare movement data. Figure 1 An image of another exemplary embodiment of a GUI for the system. DETAILED DESCRIPTION

[0046] For the purpose of promoting an understanding of the principles of the embodiments described herein, reference is now made to the drawings and descriptions in the following written specification. The references are not intended to impose any limitation on the scope of the subject matter. The present disclosure also includes any changes and modifications to the illustrated embodiments, and includes additional applications of the principles of the described embodiments, as would normally occur to a person skilled in the art to which this document belongs.

[0047] In the following embodiments, for the purpose of illustration, features of the present disclosure are discussed with respect to systems used with manual or semi-automatic assembly processes, although other uses are contemplated in other embodiments. As discussed above, the efficiency of assembly processes utilizing human workers relies on the accuracy and precision of those workers' movements. Therefore, one aspect of designing and / or maintaining efficient processes is the technology of evaluating worker performance. Unlike conventional systems that consider the results of movements, systems according to the present disclosure quantitatively calculate the accuracy of the movements themselves. In addition, while conventional systems typically require important information about the process as well as training data, systems according to the present disclosure are configured to calculate the accuracy of movements without requiring any prior information about the movements, and with very few or even only one sample movement as a reference. Systems that quantitatively analyze workers' movements have many possible applications.

[0048] For example, in some embodiments, a system according to the present disclosure is used as a training tool to educate workers on the correct movements and postures for completing a process. Examples of correct movements and postures are added to the system before or during training. The worker then practices the movements and postures of the process and receives immediate and quantitative feedback on the accuracy of the worker's movements during the process.

[0049] In another example, in some embodiments, a system according to the present disclosure is used to quantitatively evaluate a worker's movements over time. Multiple movements of workers performing the same process are quantitatively compared relative to a reference movement in order to identify, for example, problematic steps in the process and inefficiencies in the process, or to compare a worker's speed and accuracy with a reference movement and / or with other workers.

[0050] Figure 1is a schematic diagram of an exemplary embodiment of a system 100 for calculating the accuracy of continuous movement data. In some embodiments, the system 100 quantitatively calculates the accuracy, i.e., the deviation from a prescribed motion or posture, and the deviation from a target time period for completing the movement. The system 100 includes: a processor 108 operably connected to a memory 120, an input device 150, and a display output device 154. As described in more detail below, during operation, the system 100 (i) receives motion capture data 128 including a reference movement and a test movement, for example from the memory 120, the input device 150, or another source, (ii) evaluates the accuracy of the test movement relative to the reference movement, and (iii) generates calculated output visualization data 136, which includes a visual cue or guide indicating the quantitative accuracy of the test movement.

[0051] In the system 100, the processor 108 includes one or more integrated circuits that implement the functionality of a central processing unit (CPU) 112 and a graphics processing unit (GPU) 116. In some embodiments, the processor 108 is a system on a chip (SoC) that integrates the functionality of the CPU 112 and GPU 116 and optionally other components including, for example, memory 120, network devices, and positioning systems into a single integrated device, while in other embodiments, the CPU 112 and GPU 116 are connected to each other via a peripheral connection device (such as PCI Express or other suitable peripheral data connection). In one embodiment, the CPU 112 is a commercially available central processing device that implements an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. The GPU 116 includes hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics. In some embodiments, the processor 108 uses the hardware functionality in the GPU 116 to execute a software program including a driver and other software instructions to accelerate the generation and display of visualizations of graphical depictions and quantitative calculations of human movement models described herein. During operation, CPU 112 and GPU 116 execute stored program instructions 132 retrieved from memory 120. Stored program instructions 132 include software that controls the operation of CPU 112 and GPU 116 to generate visualizations of graphical depictions and quantitative calculations of human movement based on the embodiments described herein. Figure 1A processor 108 is depicted that includes a CPU 112 and a GPU 116, but alternative embodiments may omit the GPU 116 because, in some embodiments, the processor 108 in the server only generates output visualization data 136 using the CPU 112 and transmits the output visualization data 136 to a remote client computing device that uses a GPU and a display device to display image data. Additionally, alternative embodiments of the processor 108 may include a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or any other suitable digital logic device in addition to or as a replacement for the CPU 112 and the GPU 116.

[0052] In system 100, memory 120 includes both non-volatile memory and volatile memory devices. Non-volatile memory includes solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when system 100 is disabled or powered off. Volatile memory includes static and dynamic random access memory (RAM) that stores program instructions 132 and data, including motion capture data 128 and visualization data 136, during operation of system 100. In some embodiments, CPU 112 and GPU 116 each have access to separate RAM devices (e.g., a variant of DDR SDRAM for CPU 112 and a variant of GDDR, HBM, or other RAM for GPU 116), while in other embodiments, CPU 112 and GPU 116 access a shared memory device.

[0053] The memory 120 stores each of the motion capture data 128, the program instructions 132, and the output visualization data 136 in any suitable format, respectively. In the memory 120, the motion capture data 128 includes raw movement data, as well as processed movement data, such as, for example, frame, step, cycle, and time information, as discussed in further detail below. The output visualization data 136 includes one or more sets of image data generated by the system 100 to produce a calculated graphical output of the motion capture data 128. In some embodiments, the processor 108 generates the output visualization data 136 using a rasterized image format such as JPEG, PNG, GIF, etc., while in other embodiments, the processor 108 generates the output visualization data 136 using a vector image data format such as SVG or another suitable vector graphics format. The visualization data 136 may also include user interface information that can be used by the system 100 to receive instructions regarding the visualization data 136, such as, for example, the stored program instructions 132, such as via a graphical user interface (GUI).

[0054] In the system 100, the input device 150 includes any device that enables the system 100 to receive the motion capture data 128, the program instructions 132, and the visualization data 136. Examples of suitable input devices include human interface inputs such as keyboards, mice, touch screens, voice input devices, etc., and motion capture devices such as, for example, cameras, visual sensors, infrared sensors, ultrasonic sensors, accelerometers, pressure sensors, etc. A non-limiting example of the input device 150 includes a pair of digital gloves worn by a user while performing cyclical motions. The digital gloves include sensors that capture the user's motion to generate the motion capture data 128 stored in the memory 120. Additionally, in some embodiments, the system 100 implements the input device 150 as a network adapter or peripheral interconnect device that receives the motion capture data 128 from another computer or external data storage device, which can be useful for receiving large collections of motion capture data 128 in an efficient manner.

[0055] In system 100, display output device 154 includes an electronic display screen, a projector, a printer, or any other suitable device that reproduces a graphical display of output visualization data 136 generated by system 100 based on motion capture data 128. In some embodiments, system 100 includes multiple display output devices 154, and different portions of output visualization data 136 are displayed on different display output devices 154.

[0056] Although Figure 1 The system 100 is depicted as being implemented using a single computing device in conjunction with a display output device 154, but other embodiments of the system 100 include multiple computing devices. For example, in some embodiments, the processor 108 generates the output visualization data 136 as one or more image data files, and the processor 108 transmits the output visualization data 136 to a remote computing device via a data network. The remote computing device then displays the output visualization data 136, and in this embodiment, the processor 108 is operable to indirectly (rather than via) Figure 1 1 (a direct connection as depicted in FIG. 1 ) is connected to a display device in a remote client computing device. In one non-limiting example, processor 108 is implemented in a server computing device that executes stored program instructions 132 to implement a web server that transmits output visualization data 136 to a web browser in a remote client computing device via a data network. The client computing device implements a web browser or other suitable image display software to display the output visualization data 136 received from the server using a display output device 154 integrated into the client computing device.

[0057] Figure 21 is a functional block diagram depicting an exemplary embodiment of functional components 200 for operation of system 100 as implemented by processor 108 with reference to programming instructions 132. The functional components may include hardware or a combination of hardware and software. In the following description, processor 108 uses the elements of system 100 discussed above to perform actions described as being performed by or with the functional components.

[0058] Box 202 is a data synchronization component, box 212 is a data collection and training component, box 214 is a data calculation component, box 216 is a report generation component, and box 218 is a graphical user interface ("GUI") component, each of which is discussed in further detail below. In system 100, processor 108 executes stored program instructions 132 to enable CPU 112 and GPU 116 to implement the functions of data synchronization component 202, data collection and training component 212, report generation component 216, and GUI 218 in conjunction with other components in system 100. As shown in Figure 2 As depicted in , system 100 receives motion capture data 128 and generates output visualization data 136 with reference to the received motion capture data 128 .

[0059] The data synchronization component 202 is configured to receive the motion capture data 128. The motion capture data 128 includes: a plurality of records representing the position of at least one tracked item or portion of an item over time. For example, in various embodiments, the motion capture data 128 includes: one or more records of the position of reference points on a body part over time or at set time intervals, sensor data acquired over time, a video stream or a video stream that has been processed using computer vision techniques, data indicating the operating state of a machine over time, etc.

[0060] In various embodiments, the motion capture data 128 is received via the data synchronization component 202 from various sources. Figure 2 In the embodiment depicted in , motion capture data 128 is received from one or more of: a video stream 204, such as a video stream from a camera; a machine signal 206, such as a signal indicating an operating state of a machine or device; a motion capture device 208, such as a sensor (optical, infrared, ultrasonic, pressure, acceleration, etc.); or a device including one or more such sensors, such as a motion capture glove, a data storage device 210 including files of stored motion capture data; and any other acceptable peripheral device 211. In some embodiments, one or more data sources are connected to system 100 via a wired connection or a hardware connection. In some embodiments, one or more data sources are connected to system 100 via a wireless connection. In some embodiments, motion capture data 128 is received via a network connection, such as the Internet.

[0061] The data synchronization component 202 is further configured to synchronize the motion capture data 128 received from different sources onto a common timeline. For example, in some embodiments, a record of the positions of reference points on a body part taken over time is aligned in time with sensor readings or other forms of motion capture data taken over time.

[0062] In some embodiments, the motion capture data 128 includes data representing more than one continuous movement. In some embodiments, the data synchronization component 202 receives multiple motion capture data sets 128, each representing a different continuous movement. The motion capture data 128 for each individual movement is synchronized with the corresponding timeline corresponding to the movement.

[0063] The data collection and training component 212 receives synchronized motion capture data 128 from the data synchronization component 202. For the purpose of calculating the accuracy of human movement, the motion capture data 128 can generally be classified into one of two categories—reference data (i.e., baseline movements) and test data (i.e., movements to be compared and quantitatively assessed for accuracy relative to the baseline movements). In some embodiments, the data collection and training component 212 is configured to classify the first received movement from the motion capture data 128 as a reference movement, and to classify any subsequently received movements as test movements.

[0064] In some embodiments, the data collection and training component 212 is configured to separate the motion capture data 128 into individual movements. In some embodiments, the data collection and training component 212 operates iteratively and / or in conjunction with the data calculation component 214 to calculate different possible separations of the motion capture data 128 into individual movements, and selects a separation based on the accuracy calculated by the data calculation component 214.

[0065] In some embodiments, one or more movements in the motion capture data 128 include one or more labels that classify the movement as a reference movement or a test movement, respectively. In some embodiments, the data collection and training component 212 is configured to receive instructions for classifying one or more movements as a reference movement or a test movement, such as from a user or from another source via the GUI component 218.

[0066] In some embodiments, the data collection and training component 212 is configured to separate the received movement into frames, whereby a "frame" corresponds to a discrete time interval. In other words, each frame of the movement includes: a portion of the motion capture data 128 corresponding to a portion of the movement that occurs during the corresponding interval of the timeline of the movement. In some embodiments, the duration of the interval corresponding to the individual frame is preset. In some embodiments, the duration of the interval corresponding to the individual frame is set based on an instruction received from, for example, a user via the GUI component 218 or another source. In some embodiments, the duration of the interval corresponding to the individual frame is set with reference to one or more characteristics of the motion capture data 128. For example, in some embodiments, the duration of the interval corresponding to the individual frame is set with reference to one or more of the duration of the reference movement, the total travel distance of the movement, the number of individual movements or gestures within the movement, the speed of the movement, etc. Typically, the same interval of the duration of the frame is used for both the reference movement and the test movement to be evaluated relative to the reference movement.

[0067] In some embodiments, such as some embodiments in which the motion capture data 128 is received from the data storage device 210 as a file of stored motion capture data, one or more movements in the motion capture data 128 are separated into frames when received by the data synchronization component 202 and / or the data collection and training component 212.

[0068] The data collection and training component 212 is further configured to map frames of the test move to corresponding frames of the reference move. As discussed in further detail below, in some embodiments, the test move and the reference move are synchronized so that frames of the test move are mapped to frames of the reference move that correspond in time, and in some embodiments, the test move and the reference move are aligned in terms of gestures and motions within the move so that frames of the test move are mapped to frames of the reference move that correspond with respect to the sequence of motions and / or gestures performed in the move.

[0069] The data calculation component 214 receives the motion capture data 128 from the data collection and training component 212 and / or the data synchronization component 202, which has been classified as a reference motion or a test motion, respectively, which has been separated into frames and has been mapped together. The data calculation component 214 is configured to calculate the accuracy of one or more test movements relative to the reference movement, as discussed in further detail below.

[0070] In some embodiments, the data collection and training component 212 and the data calculation component 214 operate iteratively and / or in conjunction with each other to map frames of test movement to reference movement and calculate the accuracy of the mapped frames. For example, as discussed in further detail below, in some embodiments, the data collection and training component 212 calculates multiple different mappings for frames of test movement to frames of reference movement, the data calculation component 214 calculates the accuracy of the test movement relative to the reference movement for each mapping, and the data collection and training component 212 selects the mapping with the highest accuracy.

[0071] The report generation component 216 receives the motion capture data 128 and the calculated accuracy of the test movement from the data calculation component 214, and is configured to generate the output visualization data 136, as discussed in further detail below. The GUI component 218 enables a user to interact with various components of the system 100 to, for example, adjust parameters for separating the movement into frames, adjust parameters for calculating the accuracy of the test movement, instruct the system 100 to add, remove or split movements, add step identifiers to the movement, and adjust how the output visualization data 136 is displayed, as discussed in further detail below.

[0072] Figure 3 is a flow chart depicting an exemplary embodiment of a method 300 for quantitatively calculating the accuracy of a test move relative to a reference move. In the following description, references to a process 300 for performing a function or action refer to the operation of a processor executing stored program instructions to perform the function or action in association with other components in a graphic display system. For illustrative purposes, references to the process 300 for performing a function or action in association with other components in a graphic display system are described above. Figure 1 and Figure 2 The system 100 and functional components are discussed to describe the process 300 .

[0073] In box 302, the processor 108 receives motion capture data via the data synchronization component 202, whereby the motion capture data includes data representing the reference movement. In box 304, the processor 108 receives additional motion capture data via the data synchronization component 202, whereby the additional motion capture data includes data representing the test movement. In box 306, the processor 108 maps the test movement to the reference movement using the data collection and training component 212. In box 308, the processor 108 calculates the accuracy of the test movement relative to the reference movement using the data calculation component 214. In box 310, the processor 108 generates output visualization data 136 with reference to the calculated accuracy of the test movement relative to the reference movement using the report generation component 216, and in box 312, the output visualization data 136 is output to the user via the display output device 154. Further aspects of the various boxes in the method 300 are discussed below.

[0074] Referring to block 302, where the processor 108 receives motion capture data including data representing a reference movement, the motion capture data 128 is received from a variety of sources in different embodiments via the data synchronization component 202, as discussed above. In some embodiments, the motion capture data 128 is received from a motion capture device 208. In some embodiments, the motion capture device 208 is a set of gloves (not shown) that, when worn by a user, is configured to transmit motion capture data 128 that represents the orientation of the user's hands given by the direction in which each hand faces the palm, the posture of each hand given by the joint angles of the joints of each hand, and the movement given by the three-dimensional linear acceleration of each hand. In some embodiments, the set of gloves is configured to connect to the system 100 via a wireless connection protocol such as, for example, Bluetooth, Wi-Fi, radio transmission, etc. In some embodiments, the set of gloves 208 includes tracking points that are trackable using an optical tracking system such as a camera or an infrared tracking system. In some embodiments, the glove 208 includes one or more controls, such as buttons or switches, configured to cause the system 100 to begin capturing motion capture data 128 transmitted by the glove 208, and to cause the system 100 to stop capturing one or more of the motion capture data 128. In some embodiments, the processor 108 is configured to store the captured motion capture data 128 in a data file on the data storage device 210, such as in response to instructions received from a user via the GUI component 218.

[0075] In some embodiments, the motion capture data 128 is received as a data file from the data storage device 210. For example, in some embodiments, the processor 108 is configured to receive a user instruction via the GUI component 218 to load the motion capture data 128 from a data file on the data storage device 210. In some embodiments, portions of the motion capture data 128 are received from different sources. For example, in some embodiments, a reference move is loaded from a data file on a data storage device, and a test move is captured using the motion capture device 208.

[0076] In some embodiments, in response to receiving the reference movement, the processor 108 is further configured to generate output visual data 136 (eg, Figure 3 310 and 312). Figure 4 An image of a GUI 400 is depicted showing a three-dimensional animation 402 representing a user's hand during a received motion, and a timeline representation of the received movement 404. In this embodiment, the received movement is the first movement received by the system 100, and the data collection and training component 212 has classified the received movement as a reference movement.

[0077] The GUI 400 includes a variety of tools that enable a user to interact with the system 100. In this embodiment, the GUI 400 includes an input move button 406 that instructs the system 100 to receive additional moves, a refresh button 408 that instructs the system 100 to re-evaluate the moves received by the system 100, a visualize button 410 that instructs the system 100 how to visualize the output visual data 136, and a weight adjustment tool 412 that instructs the system 100 as to how to weigh certain factors when calculating the accuracy of the moves received by the system 100.

[0078] Regarding Box 304 ( Figure 3 ), wherein the processor 108 receives motion capture data including data representing a test movement via the data synchronization component 202, the test movement being received in a manner similar to the reference movement, e.g., from the data storage device 210 via the motion capture device 208. In some embodiments, in response to receiving the test movement, the processor 108 is further configured to generate, using the GUI component 218, additional output visual data 136 representing the received test movement (e.g., Figure 3 310 and 312). Figure 5 An image of the GUI 400 is depicted after the system 100 has received an additional move. In this embodiment, the data collection and training component 212 has classified the additional move as a test move. Figure 5 The GUI 400 in FIG. 4 additionally includes a timeline representation 414 of the test moves.

[0079] In some embodiments, the processor 108 receives motion capture data 128 that includes a reference movement followed by one or more test movements. As discussed in further detail below, in some embodiments, the data collection and training component 212 and the data calculation component 214 are configured to separate the motion capture data 128 into individual movements in conjunction with a process for mapping the test movements to the reference movements and calculating the accuracy of the test movements.

[0080] Regarding box 306, where the processor maps the test movement to the reference movement via the data collection and training component 212, the processor 108 maps the frames of the test movement to the frames of the reference movement so that the motion capture data corresponding to the frames of the test movement can be compared with the motion capture data of the mapped frames corresponding to the reference movement.

[0081] In some embodiments, processor 108 maps the test movement to the reference movement in time. In other words, the frames of the reference movement are synchronized with the frames of the test movement so that the start of each movement occurs at the same time. Figure 5, the timeline representation 414 of the test moves is aligned in time with the timeline representation 404 of the reference moves. In other words, the vertically aligned frames in the timeline representation 414 of the test moves and the timeline representation 404 of the reference moves occur simultaneously with respect to the start time of each move, and the timelines 404 and 414 proceed at the same rate of 1 frame per frame time interval.

[0082] As discussed above, the mapping between the reference movement and the test movement enables comparison between frames occurring at the same point in time relative to the start of the movement. In other words, the movement, posture, orientation, and timing of the test movement are directly comparable to the movement, posture, orientation, and timing of the reference movement. Figure 5 4, cursor 500 is used to select a particular frame by navigating to a horizontal position along timelines 404 and 414, causing GUI 400 to display a first three-dimensional model 504 depicting the position and pose of a reference movement at a time index corresponding to the selected frame and a second three-dimensional model 506 depicting the position and pose of a test movement at a time index corresponding to the selected frame, so as to enable a visual comparison between the reference movement and the test movement at different times along timelines 404 and 414. In some embodiments, models 504 and 506 may be played as a video stream to compare the reference movement and the test movement over time.

[0083] Refer to box 308 ( Figure 3 ), wherein the processor 108 uses the data calculation component 214 to calculate the accuracy of the test move relative to the reference move, and the processor 108 compares the motion capture data of the frame corresponding to the test move with the motion capture data of the mapped frame corresponding to the reference move. In some embodiments, the comparison performed by the processor 108 is performed between one or more of the position, orientation, posture, and movement of the user's hand for each mapped frame along the timelines 404 and 414. In other words, for each mapped frame, the processor 108 calculates the error in the reference movement relative to the difference in one or more of the position, orientation, posture, and movement of the user's hand between the test move and the reference move. The calculated error for each mapped frame represents a quantitative measurement of the accuracy of the test move relative to the reference movement for each frame along the timeline 414 of the test move.

[0084] In some embodiments, the processor 108 calculates the total error for each frame as a weighted combination of differences in one or more of the position, orientation, posture, and motion of the user's hand. In some embodiments, the processor 108 assigns relative weights to the differences based on the settings of the weight adjustment tool 412 of the GUI 400. For example, the error in each of the position, orientation, posture, and motion of the user's hand can affect the final results of different processes with different magnitudes. In some processes, the orientation of the user's hand may have a small effect on the final result, while the posture of the user's hand has a relatively large effect. In some processes, the motion of the user's hand may have a small effect, while the orientation has a relatively large effect.

[0085] With respect to block 310, where the processor 108 generates output visualization data 136 via the report generation component 216 with reference to the calculated accuracy of the reference movement relative to the test movement, and with respect to block 312, where the output visualization data 136 is output to a user, for example, via the GUI component 218, the processor 108 encodes the calculated accuracy of the test movement relative to each frame of the reference movement into the visual representation of the reference movement. Figure 5 , in this embodiment, the accuracy of each frame is encoded as the coloring and shading of the timeline representation 414 of the test move, as discussed in further detail below. Other visual indications of accuracy are used in other embodiments, such as numerical labels, pattern or image fills, charts, etc. In this embodiment, the coloring and shading from green to yellow to red indicates the frames along the timeline 414 where the test move is more accurate, moderately accurate, and less accurate relative to the reference move, respectively. Other coloring and shading gradients are used in other embodiments.

[0086] In some embodiments, processor 108 is configured to perform additional calculations to test the accuracy of the movement, such as in response to changes in calculation parameters or instructions received from a user via GUI component 218 . Figure 6a An image of a GUI 400 is depicted depicting output visualization data 136 of a plurality of test moves 602 relative to a reference move 604 calculated by the processor 108 given the particular parameters of the settings of a weight adjustment tool 412 of the GUI 400. By adjusting the settings of the weight adjustment tool 412, and activating the refresh button 408, the user can instruct the system 100 to recalculate the accuracy of the test moves 602 based on the adjusted settings. Figure 6b It illustrates how the accuracy of the calculated test moves 602 changes given different settings of the weight adjustment tool 412 ′.

[0087] Although the accuracy of the test move relative to the reference move is calculated when timelines 404 and 414 are synchronized with each other in time ( Figure 5 ) provides a direct comparison between a reference move and a test move at any given point in time, but such calculations may not be robust with respect to desynchronization between the reference move and the test move. As used herein, "desynchronization" between a reference move and a test move refers to an instance in which the reference move includes a performance of motion and gesture similar to the reference move, but on a different time scale. A reference move that performs the motion and gesture of the reference move exactly, but does so slightly faster or slower, appears to have lower accuracy under direct temporal comparison due to desynchronization of the motion and gesture between the two moves. Therefore, a computational technique that is robust to different movement rates would be beneficial.

[0088] Again, regarding box 306 ( Figure 3 ), wherein the processor maps the test movement to the reference movement via the data collection and training component 212, in some embodiments, the processor 108 is configured to map the movement based on similar motions and postures, regardless of whether the test movement and the reference movement are desynchronized, so that similar motions and postures appear at different times. In other words, the processor 108 is configured to map frames of the test movement to frames of the reference movement, the frames of the reference movement corresponding to the sequence of motions, postures and / or orientations performed in the movement.

[0089] In some embodiments, the processor 108 determines the correspondence between desynchronized motions, postures, and / or orientations via semantic analysis. In other words, since a specific motion may occur at different times in a reference motion and a test motion, in some embodiments, the processor 108 is configured to classify the motion capture data in a frame into a specific motion, posture, and / or orientation, and map the frames of the test movement to the frames of the reference movement based on similar classifications between the frames. However, classifying the motion capture data into specific motions, postures, and / or orientations may not only be computationally intensive, but may also require training data, which may be expensive and time-consuming to assemble itself. Therefore, a technique for determining the correspondence between desynchronized motions, postures, and / or orientations would be beneficial, which is agnostic about the semantics of the motions, postures, and / or orientations.

[0090] In some embodiments, processor 108 computes the mapping between frames via a dynamic time warping ("DTW") alignment process, which is a technique for comparing sequences that may differ in speed by computing the best match between discrete portions of the sequences. DTW includes computing a "warp path" that corresponds to a mapping between a portion of one sequence and a similar portion of another sequence that occurs at a different time or rate.

[0091] Figure 7700 is a visual depiction of an exemplary embodiment of a processor 108 that uses DTW to determine a warp path mapping for a test move 702 that best aligns the motion, posture, position, and orientation in the test move 702 with the motion, posture, position, and orientation in a reference move 704. The processor 108 uses DTW to compute the mapping between the reference move 704 and the test move 702 without requiring semantic knowledge of the motion, posture, position, and orientation in the timeline. Without a priori knowledge of the semantics of the timeline, the processor 108 treats the endpoints of the test move 702 and the rates at which the motion, posture, position, and orientation in the test move are effected as unknown.

[0092] like Figure 7 As shown in , the processor 108 prepares a plurality of trials 706a-d, each trial including a different selection of frames for the test moves as endpoints, and thus including a different warping path between moves. Figure 7 In FIG. 4 , four trials are depicted, but in other embodiments, any number of trials may be used. Figure 7 Also depicted are quantified values ​​of the average motion difference between reference move 704 and test move 702 given the warp path for each trial. It should be understood that, as used herein, the "warp" of DTW does not result in a modification of the rate of the test move or reference move. Rather, "warp" in this context means the calculation of a warp path that aligns desynchronized frames with similar motion, posture, and / or orientation data.

[0093] For each trial 706a-d, the processor 108 determines an optimal mapping of the frames of the test movement 702 to one or more frames of the reference movement 704 that preserves the sequential order of the mapped frames in each movement relative to each other. Preserving the sequential order means that the warp path of the mapping progresses monotonically. For example, in the first trial 706a, when the first frame 708 of the test movement 702 is mapped to the first frame 710 of the reference movement 704, the second frame 712 of the test movement 702 is mapped to the fourth frame 714 of the reference movement 704. Although the second mapping between frames 712 and 714 is not temporally aligned (i.e., frame 714 of the reference movement is temporally displaced by two frames from frame 712 in the test movement), this sequence of the first mapped frames 708, 710 preceding the second mapped frames 712, 714 is preserved. As depicted in experiments 706b-d, while each frame of test movement 702 is mapped to exactly one frame of reference movement 704, in some mappings, more than one frame of test movement 702 is mapped to a single frame of reference movement 704. Additionally, as depicted in experiments 706a and 706b, in some mappings, not all frames of reference movement 704 are mapped to frames of test movement 702.

[0094] Here, the best mapping of a trial to a given ending frame means the mapping with the lowest average error relative to the reference move among the possible mappings from that trial. For each mapping, the processor 108 calculates the average error of the trial by averaging the individual errors for each frame in the test move 702 of the trial given that particular mapping. The error for each frame of the test move, given a particular mapping, is calculated as the difference between one or more of the motion, orientation, and / or posture in the frame of the test move and the frame of the reference move mapped to that frame. Thus, for each trial 706a-d, Figure 7 The warp paths and mean motion differences depicted in correspond to the best mapping of the test moves 702 given the particular end frame of each trial 706a-d.

[0095] Processor 108 selects the trial with the lowest average error as the trial with the closest correspondence between the motion, orientation, and posture in test movement 702 and reference movement 704. Figure 7 , trial 706c has the lowest average motion difference score, which corresponds to the lowest error level in the observed test motion sequence, and therefore, has the closest correspondence between movements 702 and 704. By selecting trial 706c, processor 108 also determines the best ending frame for test movement 702 that corresponds to the end of reference movement 704.

[0096] The warp path determined for test movement 702 indicates the relative speed of the two movements. Instances where more than one frame in test movement 702 is mapped to the same frame in reference movement 704 indicate a portion of test movement 702 that is performed slower than the reference movement, and instances where frames of reference movement 702 are not mapped to frames of test movement 704 indicate a portion of test movement 702 that is performed faster than the reference movement 704. In addition, the mapping between frames is shifted in time (e.g., Figure 7 Instances of distorted path lines in are tilted) indicate similar motions, postures, and / or orientations, and although desynchronized, they are also mapped together.

[0097] Additionally, refer again to Figure 3 308, wherein the processor 108 calculates the accuracy of the test move relative to the reference move, the error for each frame of the test move being given by the error for each frame of the selected trial. Thus, the error for each frame of the test move is a quantitative assessment of the accuracy of the test move relative to the reference move that is robust to portions of the test move being performed at a rate different from the reference move.

[0098] Refer again Figure 3In block 310 of FIG. 1 , the processor 108 generates output visualization data 136 via the report generation module 216, and referring again to block 312, in block 312, the processor 108 outputs the output visualization data 136 to the user via the GUI component 218. In some embodiments, the processor 108 encodes the calculated accuracy of the test movement relative to each frame of the reference movement into the visual representation of the reference movement in the same manner as discussed above. In other words, in some embodiments, the processor 108 depicts the timeline of the test movement and the reference movement as being synchronized in time, similar to Figure 5 , Figure 6a and Figure 6b However, such a depiction may not take into account potential desynchronization between the test move(s) and the reference move. Therefore, a visual depiction of the test moves that takes into account the desynchronization would be beneficial.

[0099] As noted above, GUI 400 includes visualization buttons 410 that enable a user to select different visualizations of the test moves. Figure 8 Depicted is a GUI 400 having a visualization 800 selectable via a visualization button 410 that visually accounts for potential desynchronization between the test move(s) and the reference move. Figure 8 , the processor 108 has compressed and expanded a visualization of a timeline of test moves 802 with reference to a warp path map determined via DTW. Portions of the timeline 802 (such as region "A") where the test moves are performed faster than corresponding portions of the reference moves are stretched horizontally so that the portions are aligned horizontally, despite the difference in rate. The height of such portions is also reduced corresponding to the amount of horizontal stretching. Portions of the timeline 802 (such as region "B") where the test moves are performed slower than corresponding portions of the reference moves are compressed horizontally so that the portions are aligned horizontally, despite the difference in rate. The height of such portions is also increased corresponding to the amount of horizontal compression. As a result, an encoding of the rate at which the portion of the test moves is performed is encoded into the visualization 800 along with the accuracy of each frame.

[0100] For example, the nominal green coloring of portion "A" indicates that the motion, posture, and / or orientation of that portion of the test move was accurately performed relative to the reference move despite being performed faster (the timeline is thinning). In another example, the nominal orange and yellow coloring of portion "B" indicates that the motion, posture, and / or orientation of that portion of the test move was not performed as accurately as the reference move despite being performed slower (the timeline is thickening).

[0101] Further references Figure 3In blocks 302 and 304 of , where the motion capture data 128 is received by the system 100, in some embodiments, the movements to be evaluated are "cycles," i.e., movements that are repeated one after another. For example, in many manufacturing processes, a worker will perform a set of steps, and upon completion of the set, will repeat the set from the beginning. Therefore, in some embodiments, the motion capture data 128 received by the system 100 includes data representing more than one cycle of movement. Figure 9a The GUI 400 is depicted after the system 100 has received a reference move 902 and a multi-move loop 904. In this embodiment, the multi-move loop 904 is a sequential combination of two moves, one performed directly after the other.

[0102] In some embodiments, the GUI 400 enables the user to manually split the multi-move loop into individual moves. For example, the user can use the cursor 500 to view the frames along the multi-move loop 904 and identify the ending frame of the first move within the multi-move loop 904. Figure 9a , the user selects the identified ending frame and instructs the system 100 (in this embodiment via menu selection 906 ) to split the multi-move loop 904 at the selected frame. Figure 9b Depicted is the GUI 400 after the multi-move loop 904 has been split into two separate moves 908 and 910. Note that the scale of the timeline is Figure 9a and Figure 9b was changed to better fit the split data.

[0103] However, in some embodiments, a multi-move loop may include a large number of moves. In some embodiments, it may also be difficult to visually identify the exact frame that separates one move from the next. Therefore, a system that automatically identifies and separates moves in a multi-move loop would be beneficial.

[0104] Further reference is made to block 308 ( Figure 3 ), where the processor 108 calculates the accuracy of the test move, as discussed above, in the instance where the processor 108 uses DTW to calculate the mapping between the test move timeline and the reference move timeline, the processor 108 considers the endpoint of the test move to be unknown. In some embodiments, the frames of the test move timeline are selected as endpoints, although there are additional frames after the selected endpoint. In some embodiments where the test move timeline is a multi-move loop, the frames selected as endpoints also indicate the frames that separate one move from the next continuous move. In some embodiments, the processor 108 is configured to split the test move timeline based on the endpoint frames determined via DTW. Such splitting results in the moves that are mapped to the reference moves as discussed above, as well as the remaining moves.

[0105] In some embodiments, the processor 108 continuously performs DTW on the remaining moves to split the continuous moves within a multi-move cycle. In this way, the processor 108 is configured to split the continuous motion capture data 128 including multiple moves performed one after another into separate moves, wherein separate accuracies are calculated in the manner discussed above.

[0106] Each move split from the multi-move cycle also has a corresponding average error calculated in the manner discussed above. In some embodiments, the processor 108 is further configured to calculate confidence in the split of the multi-move cycle, whereby "confidence" means how likely it is that a given split will accurately separate two moves from each other. In some embodiments, the confidence is calculated by measuring the slope (i.e., trend) of the error at adjacent frames at a given split position. In some embodiments, the error used to compare the moves in the cycle and to calculate the confidence of the split is the error between the motion of the test move and the reference move.

[0107] With further reference to blocks 310 and 312 ( Figure 3 ), wherein the processor 108 generates output visualization data 136 and outputs the data 136 to a user, and in some embodiments, the processor 108 is further configured to generate output visualization data 136 describing additional aspects of the calculated accuracy.

[0108] In some embodiments, processor 108 is configured to generate output visualization data 136 describing multiple move cycles. Fig.10 Describe the chart 1000 of moving time, average moving error and splitting confidence, this splitting confidence is calculated for the motion capture data obtained for the cycle implemented in 3500 seconds, wherein the horizontal axis is the overall time, and the vertical axis is the time of individual movement. In chart 1000, the movement that meets the reference movement is colored as green, and the movement that does not meet the reference movement is colored as purple. In certain embodiments, processor 108 is configured to classify the movement with relatively low error and relatively high confidence as meeting, and the movement with relatively high error and relatively low confidence is classified as not meeting. Chart 1000 enables users to analyze the accuracy of a large number of cycles in batches at one time. For example, the color coding of each movement allows users to quickly identify as not meeting the specific movement, thus then the accuracy of this individual movement can be quantitatively analyzed in the above manner, to identify the specific movement, posture, position and / or orientation that causes the movement that does not meet.

[0109] Fig.11A graph 1100 depicts consecutive move times taken over a period of approximately one hour, with overall time on the horizontal axis and time for individual moves on the vertical axis. In graph 1100, compliant moves are colored blue and non-compliant moves are colored red. In some embodiments, processor 108 is configured to classify moves with an average error below a predetermined threshold as compliant and moves with an average error at or above the predetermined threshold as non-compliant. Graph 1100 allows a user to quickly identify non-compliant moves from a large batch of data.

[0110] In some embodiments, the processor 108 is further configured to generate output visualization data 136 describing the semantics of the movement. As noted above, the processor 108 is configured to calculate the accuracy of the test movement in a manner that is agnostic to the semantics of the movement, posture, position and / or orientation in the movement. However, the semantic tags of the individual movement, posture, position and / or orientation in the movement may be useful in identifying the reason why the movement deviates from the reference movement. Therefore, a system that enables semantic comparison of the movement will be beneficial. A system that can calculate the movement accuracy agnostic about semantics while still enabling semantic comparison will also be beneficial.

[0111] Fig.12 The GUI 400 is depicted after the system 100 has received a reference movement 1202. The GUI 400 enables a user to select a particular frame in the timeline of the movement 1202 and add a keyframe label to the selected frame, such as keyframe labels 1204a-c. In different embodiments, the keyframe labels indicate a particular step within the movement, a particular gesture within a step, or other information related to the movement. For example, a cycle for a manufacturing assembly line may include continuously repeated movements that each include a series of steps such as: Step 1 - "Pick up the screw from the top of the screw with your left hand, with all fingers touching the screw"; Step 2 - "Grab the syringe from the side of the syringe with your right hand", etc., and the keyframe labels can be used to identify frames in the reference movement that correspond to the start of a particular step, the completion of a step, a point of interest during a step, etc. Fig.12 In the embodiment, the key frame labels 1204a-c each include an image of the model 504 of the reference motion at the corresponding frame. In other embodiments, the key frame labels include other visualizations, such as, for example, step name labels and / or descriptions, timestamps, etc.

[0112] Typically, at least a portion of the semantics of the reference move is known. In many cases, the reference move is performed by an experienced worker or other individual who understands the correct motion, posture and orientation of the move and the important steps in the process corresponding to the move. It would also be beneficial to identify the semantics of the test move. For example, if a certain step in the process is important, dangerous or difficult, it would be beneficial to be able to highlight the step when training a new worker, evaluate how the step is implemented in particular relative to the rest of the process, or other similar tasks. However, the test move may or may not be accurate, and may or may not be implemented by a user who has a priori understanding of the semantics of the move. In addition, in instances where a large number of test moves are evaluated, manually entering the key frames of each move may be time-consuming and complicated. Therefore, a system that does not require manual entry of the semantics of the test move would also be beneficial.

[0113] In some embodiments, the processor 108 is configured to propagate the keyframe labels 1204 from the reference move to the test move received by the system 100 with reference to the mapping calculated for the test move to the reference move (block 306 ). Fig.13 A visualization is depicted of the processor 108 propagating keyframe labels from a reference move to a test move. The steps for the reference move have been added by the user in the manner discussed above, whereby a first keyframe label 1302 indicates the start of step 1, a second keyframe label 1304 indicates the start of step 2, a third keyframe label 1306 indicates the start of step 3, and a fourth keyframe label 1308 indicates the end of the move.

[0114] Referring to the mapping 1310 between the reference move and the test move, the processor 108 is configured to identify frames in the test move that correspond to the key frame labels 1302-1308 in the reference move, and propagate the key frame labels to the identified frames in the test move. In the case where the key frame label is added to a frame in the reference move that is not mapped to the test move, in some embodiments, the processor 108 is configured to propagate the key frame label to nearby frames that are mapped to the test move. In the case where the key frame label is added to a frame in the reference move that is mapped to multiple frames in the test move, in some embodiments, the processor 108 is configured to propagate the key frame label to the first (leftmost) mapped frame in the test move in chronological order.

[0115] Thus, in some embodiments, the processor 108 is configured to propagate keyframe or step labels from a reference move to a test move without manually identifying any frames in the test move. The mapping between the semantic information of the move and the reference move is used to compute semantic information about the test move without requiring semantic analysis of the test move.

[0116] Fig.14GUI 400 is depicted whereby keyframe labels in a reference move 1404 have been propagated to a plurality of different test moves 1406. Fig.14 In some embodiments, the semantic information of the test movement calculated by propagating the keyframes from the reference movement 1404 to the test movement 1406 is used to determine additional information and / or trends in the test movement 1406. For example, in Fig.14 4, the portion of the test move 1406 within the third step 1408 is illustrated as being yellower, and therefore less accurate, than other portions of the test move 1406. This indicates that the third step 1408 may be more problematic than other steps in the process. Information such as average step time, problematic steps, etc. is determined in a similar manner via visual inspection of the GUI 400.

[0117] In some embodiments, different visualizations are used to visually display additional information and trends for steps in the process. Fig.15 GUI 400 is depicted with a different visualization (e.g., accessible via visualization button 410 of GUI 400) in which the moves have been broken down so that the keyframe labels in each move are left aligned with each other. The visualization enables the user to compare step-time-variance, i.e., the change in time to complete a particular step, which may help identify steps that take too long for a worker to complete, or steps that may require more or less time than previously allotted. The visualization also enables the user to extract trends from the performance of a particular step. For example, Fig.15 Region "A" in visualizes the trend of workers taking increasingly longer to complete step 4, while exhibiting a decreasing accuracy of that step over time. Fig.16 Depicted is a GUI 400 with an additional visualization in which the timeline of movement has been warped with reference to the alignment computed during DTW by the processor 108. This visualization enables the user to identify problem steps where the worker is slow or inaccurate, as well as steps where the worker is accurate and / or efficient.

[0118] exist Fig.16 , cursor 500 is used to select a frame in a movement with high error in order to identify the difference in the posture and orientation of the worker's hand as depicted in model 504 of reference movement and model 506 of the test movement selected at that frame. Direct comparison of model 504 of reference movement and model 506 of the selected test movement enables the user to visually identify when the worker's movement in the process deviates from the reference movement, the nature of the deviation, and the impact on time and process accuracy. In other words, comparing models 504 and 506 during portions with low accuracy can provide direct visual feedback to the worker, which facilitates training of the process. For example, Fig.16The cursor 500 in FIG. 5 is to select a frame in the step that is visually indicated as having low accuracy relative to the reference movement. By comparing models 504 and 506, the user can visually determine how the motion and posture of the process can be changed in order to correct the deviated action. Fig.16 , the right hand in model 506 of the test move is depicted as being rotated clockwise relative to the right hand in model 504, and the left hand in model 506 is tilted downward relative to the left hand in model 504. Thus, a user comparing models 504 and 506 can visually determine that during this portion of the process, the left hand should be tilted more upward and the right hand should be rotated more counterclockwise relative to model 506.

[0119] In an exemplary use case of the system 100, an experienced worker records a movement of a manufacturing task, and an inexperienced worker thereafter records a performance of the same movement. The inexperienced worker then uses the system 100 to compare their performance with the movement recorded by the experienced worker in order to identify postures, movements, orientations, and / or locations in the performance that are inconsistent and / or too slow. The inexperienced worker also uses the system 100 as a guide to learn the correct posture, movement, orientation, and / or location of the movement.

[0120] In another exemplary embodiment, the system according to the present disclosure includes: a multi-threaded data receiving and synchronization component, a posture recovery component, a motion analysis and recognition component, and a user interface component. The user interface component includes a gesture visualization interface, a machine operation interface, and a data analysis interface.

[0121] It will be appreciated that variations of the above and other features and functions or alternatives thereto may be desirably combined into many other different systems, applications or methods. Various currently unforeseen or unanticipated substitutions, modifications, changes or improvements may subsequently be made by those skilled in the art, which are also intended to be covered by the present disclosure. Although several of the embodiments discussed above relate to manufacturing and assembly line production, the above variations are also applicable to other uses, such as, for example, training for recreational activities such as sports, dance, martial arts, medical uses such as physical therapy, and any other acceptable uses.

Claims

1. A method for determining the quantitative accuracy of a test movement relative to a reference movement, comprising: receiving, with a processor, motion capture data including a reference movement and a test movement; aligning the test movement with the reference movement using a processor; calculating, using a processor, a quantitative accuracy of the test movement relative to the reference movement; generating, using a processor and a display output device, a visualization representing the calculated quantitative accuracy, the visualization representing the calculated quantitative accuracy comprising an accuracy timeline; as well as encoding the calculated quantitative accuracy as the coloring and shading of the accuracy timeline to illustrate the corresponding calculated quantitative accuracy value at each time point of the accuracy timeline, Wherein the processor is further configured to warp the accuracy timeline with reference to a warping path generated by a dynamic time warping process, the warping comprising: Thickening and compression represent portions of the accuracy timeline for portions of the test move that are performed slower relative to corresponding portions of the reference move, the quantitative amounts of thickening and compression being based on the relative rates between the corresponding portions; and Thinning and stretching represent the portion of the accuracy timeline for the portion of the test move that is performed faster relative to the corresponding portion of the reference move, and the quantitative amount of thinning and stretching is based on the relative rates between the corresponding portions.

2. The method according to claim 1, wherein: The reference movement is represented as a first sequence of frames each including motion capture data corresponding to a respective time interval of the first sequence; The test movement is represented as a second sequence of frames each including motion capture data corresponding to a respective time interval of the second sequence; as well as Aligning the test move with the reference move includes mapping each frame of the second sequence for the test move to at least one frame of the first sequence for the reference move such that a sequence order of the mapped frames in each of the reference move and the test move relative to each other is preserved.

3. The method according to claim 2, wherein: The frames of the second sequence for the test move are mapped to the frames of the first sequence for the reference move via a dynamic time warp alignment process that determines a warp path to align desynchronized portions of the test move and the reference move.

4. The method according to claim 3, further comprising: generating, with a processor and a display output device, a visualization representing the reference movement, the visualization comprising a reference timeline formed by frames of the first sequence; Therein, the accuracy timeline is based on frames of the second sequence.

5. The method according to claim 1, wherein: The motion capture data includes a record describing at least one of a motion, a posture, an orientation, and a position performed over time for each of the test move and the reference move; as well as The quantitative accuracy of the test movement relative to the reference movement is calculated with reference to a quantitative difference over time of at least one of motion, posture, orientation, and position between the test movement and the reference movement.

6. The method according to claim 1, wherein: The motion capture data comprises a plurality of individual test moves including the test move.

7. The method according to claim 6, wherein: The plurality of individual test moves are in the form of a continuous multi-move cycle; as well as The processor is further configured to split the continuous multi-move cycle into individual test moves when aligning the test move with the reference move.

8. The method according to claim 6, wherein: The processor is further configured to: calculating the respective quantitative accuracy of each individual test movement relative to the reference movement; and A visualization representing each individual test movement is generated using a display output device, wherein the calculated quantitative accuracy of the test movement relative to the reference movement is encoded in the visualization representing each individual test movement.

9. A system for determining the quantitative accuracy of a test movement relative to a reference movement, comprising: Display output devices; A memory configured to store: Programming instructions; as well as Motion capture data, including: Reference movement; and Test moves; and a processor operably connected to the display output device and the memory and configured to execute programmed instructions to determine a quantitative accuracy of the test movement relative to the reference movement, wherein the determination of the quantitative accuracy comprises: aligning the test move with the reference move; calculating the quantitative accuracy of the test movement relative to the reference movement; generating, using a display output device, a visualization representing the calculated directional accuracy, the visualization representing the calculated quantitative accuracy comprising an accuracy timeline; and encoding the calculated quantitative accuracy as the coloring and shading of the accuracy timeline to illustrate the corresponding calculated quantitative accuracy value at each time point of the accuracy timeline, The processor is further configured to warp the accuracy timeline with reference to a warping path determined via a dynamic time warping process, the warping comprising: Thickening and compression represent portions of the accuracy timeline for portions of the test move that are performed slower relative to corresponding portions of the reference move, the quantitative amounts of thickening and compression being based on the relative rates between the corresponding portions; and Thinning and stretching represent the portion of the accuracy timeline for the portion of the test move that is performed faster relative to the corresponding portion of the reference move, and the quantitative amount of thinning and stretching is based on the relative rates between the corresponding portions.

10. The system of claim 9, wherein: The reference movement is represented as a first sequence of frames, each of which includes motion capture data corresponding to a corresponding time interval of the first sequence; the test movement is represented as a second sequence of frames, each of which includes motion capture data corresponding to a corresponding time interval of the second sequence; as well as Aligning the test move with the reference move includes mapping each frame of the second sequence for the test move to at least one frame of the first sequence for the reference move such that a sequence order of the mapped frames in each of the reference move and the test move relative to each other is preserved.

11. The system according to claim 10, wherein: The processor is further configured to map the second sequence of frames for the test move to the first sequence of frames for the reference move via a dynamic time warp alignment process that determines a warp path to align desynchronized portions of the test move and the reference move.

12. The system of claim 11, wherein: The determination of the accuracy further comprises generating, using a display output device, a visualization representing the reference movement, the visualization comprising a reference timeline formed by frames of the first sequence; and The accuracy timeline is based on frames of the second sequence.

13. The system of claim 9, further comprising: A motion capture device is configured to transfer motion capture data to a memory.

14. The system according to claim 13, wherein: The motion capture device is a pair of motion capture gloves.

Images