Repetitive human activity abnormal motion detection

Abstract

The invention relates to repetitive human activity anomaly motion detection. In particular, anomaly motion is detected in sensor data collected about a performance of a repetitive human activity. An autoencoder network model is trained based on a set of standard activities. A repetitive activity is extracted from the sensor data. A first score is generated that indicates how far the repetitive activity is from the standard activities. The repetitive activity is used to retrain the autoencoder network model using weights of the autoencoder network model as initial values based on the training of the autoencoder network model using the set of standard activities. A second score is generated that indicates whether the repetition is an outlier compared to other repetitions of the repetitive activity. A final score is generated based on a weighting of the first score and the second score.

Description

Technical Field

[0001] This disclosure relates to the detection of anomalous motion in sensor data, wherein the sensor data is collected regarding the performance of repetitive human activities. Background Technology

[0002] With the proliferation of low-cost wearable sensors such as accelerometers, gyroscopes, and magnetometers, large-scale human motion signals have become widely available. To analyze motion data and further understand the human activities underlying it, it is necessary to develop intelligent systems that are highly scalable and robust to various real-world scenarios.

[0003] One type of scenario from which such data can be learned is motion signal analysis, where the activities considered are inherently repetitive and periodic. This type of data is common in many fields, such as exercise routines for fitness, assembly operations for manufacturing, and checkout data from cashiers in grocery stores. Summary of the Invention

[0004] According to an illustrative example, a method for detecting anomalous motion in sensor data collected regarding the performance of repetitive human activities includes: training an autoencoder network model based on a standard activity set; extracting repetitive activities from the sensor data; generating a first score indicating the distance of a repetition of the repetitive activity from the standard activities; retraining the autoencoder network model using the repetitive activities, using the weights of the autoencoder network model as initial values ​​based on training the autoencoder network model using the standard activity set; generating a second score indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and generating a final score based on the weights of the first and second scores.

[0005] According to an illustrative example, a system for detecting anomalous motion in an activity sequence includes a display device, a memory configured to store motion analysis applications, and a processor. The processor is operatively connected to the memory and the display device and configured to execute the motion analysis applications to: train an autoencoder network model based on a standard activity set; extract repetitive activities from sensor data; generate a first score indicating the distance of a repetition of a repetitive activity from a standard activity; retrain the autoencoder network model using the repetitive activities, using the weights of the autoencoder network model as initial values, which are based on the autoencoder network model trained using the standard activity set; generate a second score indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and generate a final score based on the weights of the first and second scores.

[0006] According to an illustrative example, a non-transitory computer-readable medium includes instructions for a motion analysis application that, when executed by one or more processors, cause the one or more processors to: train an autoencoder network model based on a standard activity set; extract repetitive activities from sensor data; generate a first score indicating the distance of a repetition of a repetitive activity from the standard activities; retrain the autoencoder network model using the repetitive activities, using the weights of the autoencoder network model as initial values, the weights being based on the autoencoder network model trained using the standard activity set; generate a second score indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and generate a final score based on the weights of the first and second scores.

[0007] The present invention also provides the following technical solutions:

[0008] 1. A method for detecting anomalous motion in sensor data collected regarding performance of repetitive human activities, comprising:

[0009] Train an autoencoder network model based on a standard activity set;

[0010] Extracting repetitive activities from sensor data;

[0011] Generate a first score, which indicates the distance between the repetition of the repetitive activity and the standard activity;

[0012] The repetitive activities are used to retrain the autoencoder network model, using the weights of the autoencoder network model as initial values, the weights being based on training the autoencoder network model using the standard activity set;

[0013] A second score is generated, indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and

[0014] The final score is generated based on the weights of the first score and the second score.

[0015] 2. The method according to Scheme 1 further includes:

[0016] Each of the multiple sensor inputs of the sensor data is applied to a corresponding one of the multiple encoder subnetworks of the encoder in the autoencoder network model;

[0017] The outputs of the multiple encoder subnetworks are applied to another encoder subnetwork of the encoder to fuse the multiple sensor inputs to generate the extracted motion window output; and

[0018] The extracted motion window output is applied to the decoder of the autoencoder network pattern, the decoder including a set of decoder subnetworks that mirror the plurality of encoder subnetworks, and the decoder regenerates the estimated motion window to minimize the reconstruction error separated to the plurality of sensor inputs.

[0019] 3. The method according to Scheme 2 further includes learning the parameters of the encoder subnetwork and the decoder subnetwork by backpropagation through minimizing the mean squared error loss.

[0020] 4. The method according to Scheme 2, wherein the sub-network includes one or more of a fully connected network, a convolutional neural network, a long short-term memory network, or an attention network.

[0021] 5. The method according to Scheme 1 further includes determining the weights of the first score and the second score based on one or more of the confidence level in the distance of the repetitive activity from the standard activity or the probability of sensor drift of the sensor that generates the sensor data.

[0022] 6. The method according to Scheme 1, wherein the sensor data is motion capture data captured from a wearable glove motion capture device.

[0023] 7. The method according to Scheme 1, wherein the repetitive human activity includes one or more of physical exercise, manufacturing assembly operations, or store checkout operations.

[0024] 8. A system for detecting anomalous motion in an activity sequence, comprising:

[0025] Display devices;

[0026] Memory configured for storing motion analysis applications; and

[0027] A processor, operatively connected to the memory and the display device, and configured to execute the motion analysis application to...

[0028] Train an autoencoder network model based on a standard activity set;

[0029] Extracting repetitive activities from sensor data;

[0030] Generate a first score, which indicates the distance between the repetition of the repetitive activity and the standard activity;

[0031] The repetitive activities are used to retrain the autoencoder network model, using the weights of the autoencoder network model as initial values, the weights being based on the autoencoder network model trained using the standard activity set;

[0032] A second score is generated, indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and

[0033] The final score is generated based on the weights of the first score and the second score.

[0034] 9. The system according to claim 8, wherein the processor is further configured to execute the motion analysis application to:

[0035] Each of the multiple sensor inputs of the sensor data is applied to a corresponding one of the multiple encoder subnetworks of the encoder in the autoencoder network model;

[0036] The outputs of the multiple encoder subnetworks are applied to another encoder subnetwork of the encoder to fuse the multiple sensor inputs to generate the extracted motion window output; and

[0037] The extracted motion window output is applied to the decoder of the autoencoder network pattern, the decoder including a set of decoder subnetworks that mirror the plurality of encoder subnetworks, and the decoder regenerates the estimated motion window to minimize the reconstruction error separated to the plurality of sensor inputs.

[0038] 10. The system according to claim 9, wherein the processor is further configured to execute the motion analysis application to learn the parameters of the encoder subnetwork and the decoder subnetwork by backpropagation through minimizing the mean square error loss.

[0039] 11. The system according to Scheme 9, wherein the sub-network includes one or more of a fully connected network, a convolutional neural network, a long short-term memory network, or an attention network.

[0040] 12. The system according to claim 8, wherein the processor is further configured to execute the motion analysis application to determine the weights of the first score and the second score based on one or more of the confidence level in the distance of the repetitive activity from the standard activity or the probability of sensor drift of the sensor that generates the sensor data.

[0041] 13. The system according to claim 8 further includes a wearable glove motion capture device, wherein the sensor data is motion capture data captured from the wearable glove motion capture device.

[0042] 14. A non-transitory computer-readable medium comprising instructions for a motion analysis application, wherein when one or more processors execute the instructions for the motion analysis application, the instructions for the motion analysis application cause the one or more processors to:

[0043] Train an autoencoder network model based on a standard activity set;

[0044] Extracting repetitive activities from sensor data;

[0045] Generate a first score, which indicates the distance between the repetition of the repetitive activity and the standard activity;

[0046] The repetitive activities are used to retrain the autoencoder network model, using the weights of the autoencoder network model as initial values, the weights being based on the autoencoder network model trained using the standard activity set;

[0047] A second score is generated, indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and

[0048] The final score is generated based on the weights of the first score and the second score.

[0049] 15. The medium according to claim 14 further includes instructions for the motion analysis application, which, when the one or more processors execute the instructions for the motion analysis application, cause the one or more processors to:

[0050] Each of the multiple sensor inputs of the sensor data is applied to a corresponding one of the multiple encoder subnetworks of the encoder in the autoencoder network model;

[0051] The outputs of the multiple encoder subnetworks are applied to another encoder subnetwork of the encoder to fuse the multiple sensor inputs to generate the extracted motion window output; and

[0052] The extracted motion window output is applied to the decoder of the autoencoder network pattern, the decoder including a set of decoder subnetworks that mirror the plurality of encoder subnetworks, and the decoder regenerates the estimated motion window to minimize the reconstruction error separated to the plurality of sensor inputs.

[0053] 16. The medium according to claim 15 further includes instructions for the motion analysis application, wherein when the one or more processors execute the instructions for the motion analysis application, the instructions for the motion analysis application cause the one or more processors to learn the parameters of the encoder subnetwork and the decoder subnetwork by backpropagation through minimizing the mean squared error loss.

[0054] 17. The medium according to claim 15, wherein the sub-network comprises one or more of a fully connected network, a convolutional neural network, a long short-term memory network, or an attention network.

[0055] 18. The medium according to claim 14 further includes instructions for the motion analysis application, wherein when the one or more processors execute the instructions for the motion analysis application, the instructions for the motion analysis application cause the one or more processors to determine the weights of the first score and the second score based on one or more of the confidence level in the distance of the repetitive activity from the standard activity or the probability of sensor drift of the sensor that generates the sensor data.

[0056] 19. The medium according to claim 14, wherein the sensor data is motion capture data captured from a wearable glove motion capture device. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of an exemplary embodiment of a system for detecting abnormal motion in sensor data collected regarding performance of repetitive human activities;

[0058] Figure 2 An example operational overview of the process for detecting anomalies in repetitive activities is shown;

[0059] Figure 3 Example details of a model based on an autoencoder architecture are shown; and

[0060] Figure 4 An example of an exemplary sliding window method for time segmentation is shown. Detailed Implementation

[0061] Embodiments of this disclosure are described herein. However, it will be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or reduced to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to implement the embodiments in diverse ways. As will be understood by those skilled in the art, various features shown and described with reference to any of the drawings may be combined with features shown in one or more other drawings to produce embodiments not explicitly shown or described. The combinations of features shown provide representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for particular applications or implementations.

[0062] A method for systematically identifying aberrant movements in repetitive activities is described. A key issue in analyzing repetitive activities is detecting and locating aberrant movements. Aberrant movements are characterized by being clearly distinguishable from standard movements and / or the rest of the repetitive pattern. It should be noted that aberrant movements can occur at any time and for any duration. Successfully identifying and locating aberrant movements can draw the user's attention to the anomaly and help them improve future standardization.

[0063] Understanding repetitive human activities from wearable sensor data is a crucial issue in fitness monitoring, manufacturing, retail, and other physical activities. However, anomaly detection is challenging for several reasons. One reason is that the variation in movement between repetitions can be significant. While overall repetitive patterns can remain largely unchanged, short-duration movements are often prone to change over time. These variations can easily be confused with genuine anomalies, making it challenging for algorithms to control false positives. Another challenge is that data drift is more likely to occur as activity lasts longer. Several factors can contribute to drift, including internal sensor bias, environmental noise, and muscle fatigue. In this scenario, comparisons to standard movements are no longer a valid criterion for identifying anomalies, as even normal movements can show significant differences due to drifting sensor readings.

[0064] In this disclosure, the aforementioned challenges are addressed by combining normalization evaluation and outlier detection within an autoencoder framework. The evaluation step compares each motion only to a predefined set of standard motions, while the outlier detector compares each motion to all other recorded motions to identify outliers. Each criterion contributes to identifying genuine anomalies and provides synergy. Furthermore, an autoencoder neural network is used to fuse multiple sensors and / or modalities to derive a latent motion space, where motions are represented by low-dimensional vectors, and both criteria are applied. Compared to previous methods that either consider only one criterion or operate on the raw signal, the proposed method exhibits better robustness to motion variations, data drift, and incomplete references.

[0065] Figure 1This is a schematic diagram of an exemplary embodiment of system 100 for detecting anomalous motion in sensor data collected regarding the performance of repetitive human activities. System 100 includes a processor 102 operatively connected to a display device 108, a memory 110, an input device 118, and a motion capture device 120. As described in more detail below, during operation, system 100 (i) trains an autoencoder network model 126 based on a relatively small set of standard activities 122; (ii) extracts repetitive activities 124 from sensor data (e.g., motion capture data 114 from motion capture device 120) to generate a first score 128 indicating the distance of a repetition of repetitive activity 124 from the standard activity 122; (iii) trains the autoencoder network model 126 using repetitive activities 124 and generates a second score 128 indicating whether the repetition is an outlier; and (iv) generates a final score 128 based on the weights of the first score 128 and the second score 128.

[0066] In system 100, processor 102 includes one or more integrated circuits that implement the functions of central processing unit (CPU) 104 and graphics processing unit (GPU) 106. In some examples, processor 102 is a system-on-a-chip (SoC) that integrates the functions of CPU 104 and GPU 106, along with optional other components, into a single integrated device, including, for example, memory 110, network devices, and positioning systems. In other examples, CPU 104 and GPU 106 are interconnected via peripheral connectivity devices such as PCI Express or other suitable peripheral data connections. In one example, CPU 104 is a commercially available central processing unit that implements an instruction set such as x86, ARM, Power, or MIPS instruction set families.

[0067] GPU 106 may include hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics on display device 108. Display device 108 may include an electronic display screen, projector, printer, or any other suitable device capable of reproducing the graphics display. In some examples, processor 102 uses the hardware capabilities in GPU 106 to execute software programs, including drivers and other software instructions, to accelerate the graphical depiction of human movement models and the generation and display of visualizations of quantitative computations described herein.

[0068] During operation, CPU 104 and GPU 106 execute stored program instructions retrieved from memory 110. The stored program instructions include software that controls the operation of CPU 104 and GPU 106 to perform the operations described herein.

[0069] although Figure 1 Processor 102 is described as including both CPU 104 and GPU 106; however, alternative embodiments may omit GPU 106. For example, processor 102 may be a server that uses only CPU 104 to generate output visualization data and transmit the output visualization data to a remote client computing device that uses GPU 106 and display device 108 to display the data. Furthermore, alternative embodiments of processor 102, in addition to or as a replacement for CPU 104 and GPU 106, may include a microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), digital signal processor (DSP), or any other suitable digital logic device.

[0070] In system 100, memory 110 includes non-volatile memory and volatile memory devices. Non-volatile memory includes solid-state memory, such as NAND flash memory, magnetic storage media, and optical storage media, or any other suitable data storage device that retains data when system 100 is deactivated or loses power. Volatile memory includes static and dynamic random access memory (RAM) storing program instructions and data, including motion analysis applications 112, motion capture data 114, standard activities 122, repetitive activities 124, models 126, and scores 128 during system 100 shutdown. In some embodiments, CPU 104 and GPU 106 each have access to a separate RAM device (e.g., a variant of DDR SDRAM for CPU 104 and a variant of GDDR, HBM, or other RAM for GPU 106), while in other embodiments, CPU 104 and GPU 106 access a shared memory device. The memory 110 can store motion analysis applications 112, motion capture data 114, standard activities 122, repetitive activities 124, models 126, and scores 128 for maintenance and retrieval.

[0071] Input device 118 may include any of a variety of devices that enable system 100 to receive motion capture data 114, motion analysis application 112, and standardized evaluation results 116. Examples of suitable input devices include human-machine interface inputs such as keyboards, mice, touchscreens, voice input devices, and so on. In some examples, system 100 implements input device 118 as a network adapter or peripheral interconnect device that receives data from another computer or external data storage device, which can be useful for efficiently receiving large sets of motion capture data 114.

[0072] The motion analysis application 112 includes instructions that, when executed by the processor 102 of the system 100, cause the system 100 to perform the processes and operations described herein.

[0073] Motion capture data 114 refers to multiple records representing the position of at least one tracked object or part of an object over time. For example, motion capture data 114 may include one or more of the following: position records of reference points on body parts over time or at set time intervals, sensor data acquired over time, video streams or video streams already processed using computer vision techniques, data indicating the operational status of a machine over time, etc. In some cases, motion capture data 114 may include data representing more than one continuous movement. For example, motion capture data 114 may include a combination of sets of multiple combined motion capture data 114. Motion capture data 114 may also include processed movement data, such as frame, step, period, and time information collected from raw movement data.

[0074] Motion capture device 120 is a device configured to generate motion capture data 114. As some non-limiting examples, motion capture device 120 may include: a camera, a vision sensor, an infrared sensor, an ultrasonic sensor, an accelerometer, a pressure sensor, etc. A non-limiting example of motion capture device 120 is one or a pair of digital gloves worn by a user while performing repetitive movements. The digital gloves may include multiple sensors that combine to capture multiple aspects of the user's movements to generate motion capture data 114. In this example, motion capture data 114 may be received from motion capture device 120 and stored in memory 110.

[0075] An activity is an action performed by a user. Examples of such activities include exercise, assembly operations in manufacturing, and checkout operations at a store. Standard activity 122 may refer to a baseline or specification version of the movement embodied in the activity. For example, standard activity 122 may be indicated by data indicating the correct performance of an exercise, assembly operation, or checkout operation. Standard activity 122 can be used as a comparison point with other movements. This allows identification of how close the other movement is to standard activity 122. Repetitive activity 124 may refer to such other movements and may include one or more repetitions of an activity performed by a user. If the data indicating another movement is similar to the data regarding standard activity 122, the movement is considered to be performed correctly. However, if the data differs, the movement is considered to be performed incorrectly. In the example, standard activity 122 and / or repetitive activity 124 may be observed and / or recorded by motion capture device 120.

[0076] In artificial intelligence (AI) or machine learning systems, model-based inference refers to reasoning methods based on the operation of a machine learning model 126. Typically, the machine learning model 126 is trained to learn the ability to provide precise correlations between input and output values. At runtime, the machine learning engine uses the knowledge encoded in the machine learning model 126 to arrive at conclusions such as diagnoses or predictions based on observed data. In the example described in this paper, model 126 employs an autoencoder architecture. An autoencoder is a type of machine learning model that performs unsupervised learning on data to identify relevant representations or encodings about excluding elements considered irrelevant. From this dimensionality-reduction encoding, the autoencoder also attempts to regenerate an approximation of the original input, aiming to make the regenerated output match the original input as closely as possible. Therefore, an autoencoder includes both encoder and decoder aspects.

[0077] Score 128 refers to a value calculated using model 126 that indicates the likelihood that the case of repetitive activity 124 matches the reference standard activity 122. As used herein, these scores 128 may indicate the distance of repetitive activity 124 relative to the reference standard activity 122, whether the case of repetitive activity 124 is an outlier compared to other cases of repetitive activity 124, and / or a weighted average of the two types of scores.

[0078] Figure 2 An example operational overview of a process 200 for detecting anomalies in repetitive activities is shown. (Reference) Figure 2 And continue to refer to Figure 1 System 100, process 200 includes four main operations. First, model 126 is trained based on a set of small standard activities 122. Typically, the encoder of model 126 learns to extract representations of low-dimensional motion, and the decoder of model 126 learns to reconstruct the input motion from these representations. The motion representations can be stored as a set of references to the standard activities 122.

[0079] Secondly, during use, representations of the repetitive activities 124 are extracted. In this example, these representations are extracted from data collected since the motion capture device 120. Each of the movements is compared to a reference set, and anomaly scores 128 are calculated, where larger values ​​correspond to a higher probability of anomalous movement.

[0080] Third, after recording is complete, the recorded motion (e.g., excluding the standard activity 122 reference) is used to retrain the autoencoder network model 126. The resulting motion representations are compared in pairs to identify outliers. Another outlier score 128 is calculated based on the distance to the nearest neighbor.

[0081] Finally, a motion anomaly is calculated based on a weighted parameter combination with respect to the scores 128 of the second and third operations. This final weighted score 128 can then be used to identify anomalous motions.

[0082] The details of model 126 and the operation of process 200 are discussed in further detail below. Table 1 summarizes some of the symbols used throughout the discussion of model 126 and process 200:

[0083] symbol represent Index of the motion window in standard activities Index of motion windows during repetitive activities over time. Sensor index and total number of sensors Extracted motion window The latent representation of the corresponding motion window extraction

[0084] Table 1: Symbols.

[0085] Figure 3 Example details of model 126 based on an autoencoder architecture are shown. As shown, each trapezoidal block represents a neural subnetwork 302. As mentioned above, the autoencoder model 126 includes an encoder 304 and a decoder 306. The encoder 304 also includes multiple subnetworks 302. Each subnetwork 302 of the encoder 304 is structurally mirrored to the corresponding subnetwork 302 of the decoder 306 to form the entire decoder 206. As shown on the encoder 304 side, the input includes information about... K Each extracted motion window from one of the sensors (e.g., in motion capture device 120) x .Will K Each sensor input is applied to a different one in subnetwork 302, which in turn feeds into one or more additional layers that provide the extracted motion window output. z On the decoder 306 side, the extracted motion window is output. z The feed is passed through the mirror decoder 306 to regenerate an estimated motion window that attempts to minimize the reconstruction error. .

[0086] Regarding the first main operation, namely the operation of the autoencoder model 126 trained based on standard activity 122, the following example implementation is provided. Before using system 100, the user records several cycles of standard activity 122 as a reference. Based on the recording of standard activity 122, time segmentation is then performed using a sliding window to capture short-duration motion patterns. w Represents the window length, and in... The stride representing the movement of the window can be used to illustrate the time segmentation process, such as in... Figure 4 As shown in the diagram. Since multiple sensor types and multiple placements can exist, time segmentation is performed for each existing sensor. K This represents the number of sensors, and the length of standard activity 122 is recorded as... l std The number of motion windows for each sensor as a segmentation result is: And the total number of motion windows for all sensors is .

[0087] Using motion windows as input data, an autoencoder neural network model 126, comprising encoder 304 and decoder 306, is trained, as shown in... Figure 2 As visualized in the diagram, encoder 304 comprises multiple sub-networks 302. In the illustrated example, each sensor is first processed by a separate sub-network 302, and then all sensors are fused via another sub-network 302. Sub-networks 302 can utilize any network architecture suitable for the application, including fully connected networks, convolutional neural networks (e.g., as in Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, “Imagenet classification with deep convolutional neural networks”). Advances in neural information processing systems (Discussed in 2012), Long Short-Term Memory (LSTM) networks (e.g., as in Hochreiter, Sepp, and Jürgen Schmidhuber, “Long short-term memory”). Neural computation (Discussed in 9.8 (1997): 1735-1780) and attention networks (e.g., as in Song, Huan, et al. “Attend and diagnose: Clinical time series analysis using attention models.”). Thirty-Second AAAI Conference on Artificial Intelligence (Discussed in 2018). The mirror layer then constitutes part 306 of the decoder. The overall network parameters are learned via backpropagation by minimizing the mean squared error loss described below (e.g., as in Baldi, Pierre, "Autoencoders, unsupervised learning, and deep architectures"). Proceedings of ICML workshop on unsupervised and transfer learning (As discussed in 2012), as shown in formula (1):

[0088]

[0089] in Representative about sensors k window i Reconstructed motion window at decoder output.

[0090] Continue to refer to Figure 3 After training model 126, the input window of standard activity 122 is... Feed to encoder 304, and from the bottleneck layer z i The latent representations are extracted as output. These representations contain fused motion features corresponding to the input window. Note that the following steps in the described system 100 also follow this framework, and the computation of the anomaly score 128 is performed in the latent space.

[0091] The autoencoder neural network model 126 serves a dual purpose: First, it integrates measurements from multiple sensors to derive a concise yet rich representation, which is robust against sensor noise and motion variations. Second, depending on the specific network architecture, the encoder 304 can extract different levels of time dependence, which is better suited for characterizing motion windows than the raw measurements.

[0092] After the autoencoder has been trained based on standard activity 122, a normalized evaluation can then be performed. With reference standard activity 122 provided and the first operation completed, the user can begin using system 100. In the normalized evaluation step, the model 126 trained in the first operation can be replicated, and information about each motion window can be extracted from the data stream. x j The potential representations are then compared with all representations from the reference. z i The comparison is performed, and, as shown in formula (2), the abnormality score 128 is calculated:

[0093]

[0094] The first two operations constitute a standardized evaluation criterion. As mentioned above, although this criterion is easy to implement, it may fail when sensor recordings drift or when the predefined normal motion set is incomplete. Therefore, System 100 further incorporates an outlier detection process in the next operation to address this challenge.

[0095] Continue to refer to Figure 2 The third operation involves retraining the autoencoder model 126 based on the repetitive activities 124 and outlier detection. When recording is complete, the repetitive activity 124 data can be utilized by retraining the autoencoder model 126. N j The number of motion windows in the recording can be initialized using the 126 weights from the autoencoder model of Operation 2. Then, the set can be used... Upgrade network parameters. Note that sensor indexes have been omitted for clarity. k .

[0096] The motion latent space derived from retraining enables the detection of anomalies as outliers. Outliers can be considered as samples with features that are significantly different from all other samples. Based on this intuition, for a sample... x j First, the distance to the motion window of all other records in the potential space is calculated, and then anomaly scores are defined for the instantaneous motion window based on the nearest neighbor, as shown in Equation (3):

[0097]

[0098] in M It represents the number of nearest neighbors considered (excluding itself).

[0099] Therefore, even when data drift occurs or new motions undefined in the reference occur, normal motions will still repeat due to the repetitive nature of the activity, while anomalous motions are rare and may differ each time they occur. The entire retraining and outlier detection process is... Figure 2 The diagram shows operation three.

[0100] Proceeding to step four, the outlier scores identified in step three are integrated. It's important to note that the outlier detection and normalization evaluation phases are complementary and, in some cases, indispensable. On one hand, comparing solely with the reference standard movement 122 can be problematic when data drift occurs or when definitions are incomplete. On the other hand, outlier detection cannot be the sole criterion. Consider the following corner case where a user misinterprets an outlier movement as normal and therefore repeats it multiple times. The outlier detector might identify several nearest neighbors and return low outlier scores, resulting in false negatives. Conversely, the normalization evaluator remains unaffected and is able to identify high outlier scores each time a movement occurs.

[0101] Therefore, the key lies in considering the two standards together by unifying their respective outlier scores 128. Thus, the final outlier score 128 can be calculated as shown in formula (4):

[0102]

[0103] Where parameters The two criteria are balanced and can be determined based on or with reference to the confidence level or the possibility of sensor drift in criterion activity 122.

[0104] The processes, methods, or algorithms disclosed herein can be transmitted to, implemented by, or fed to a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored in various forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media (such as ROM devices) and information variablely stored on writable storage media (such as floppy disks, magnetic tapes, CDs, RAM devices, other magnetic media, and optical media). The processes, methods, or algorithms can also be implemented in a software executable object. Alternatively, the processes, methods, or algorithms can be embodied, wholly or partially, using suitable hardware components (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices), or using a combination of hardware, software, and firmware components.

[0105] While exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms covered by the claims. The vocabulary used in this specification is descriptive rather than limiting, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously described, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing an advantage or being preferred relative to other embodiments or prior art implementations with respect to one or more desired features, those skilled in the art will recognize that compromises may be made on one or more features to obtain desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Thus, to the extent that any embodiment described with respect to one or more features is less desirable compared to other embodiments or prior art implementations, these embodiments are not outside the scope of this disclosure and may be desired for a particular application.

Claims

1. A method for detecting anomalous motion in sensor data collected regarding performance of repetitive human activities, comprising: Train an autoencoder network model based on a standard activity set; Extracting repetitive activities from sensor data; The trained autoencoder network model is used to generate a first score, which indicates the distance between the repetition of the repetitive activity and the standard activity. The repetitive activities are used to retrain the autoencoder network model, using the weights of the autoencoder network model as initial values, the weights being based on training the autoencoder network model using the standard activity set; A second score is generated using the retrained autoencoder network model, indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and The final score is generated based on the weights of the first score and the second score.

2. The method according to claim 1, further comprising: Each of the multiple sensor inputs of the sensor data is applied to a corresponding one of the multiple encoder subnetworks of the encoder in the autoencoder network model; The outputs of the multiple encoder subnetworks are applied to another encoder subnetwork of the encoder to fuse the multiple sensor inputs to generate the extracted motion window output; and The extracted motion window output is applied to the decoder of the autoencoder network pattern, the decoder including a set of decoder subnetworks that mirror the plurality of encoder subnetworks, and the decoder regenerates the estimated motion window to minimize the reconstruction error separated to the plurality of sensor inputs.

3. The method of claim 2, further comprising learning the parameters of the encoder subnetwork and the decoder subnetwork by backpropagation through minimizing the mean squared error loss.

4. The method according to claim 2, wherein, The subnetworks include one or more of fully connected networks, convolutional neural networks, long short-term memory networks, or attention networks.

5. The method of claim 1, further comprising determining the weights of the first score and the second score based on one or more of the confidence level in the distance of the repetitive activity from the standard activity or the probability of sensor drift of the sensor that generates the sensor data.

6. The method according to claim 1, wherein, The sensor data is motion capture data captured from a wearable glove motion capture device.

7. The method according to claim 1, wherein, The repetitive human activities include one or more of the following: physical exercise, manufacturing assembly operations, or store checkout operations.

8. A system for detecting anomalous motion in an activity sequence, comprising: Display devices; Memory configured for storing motion analysis applications; and A processor, operatively connected to the memory and the display device, and configured to execute the motion analysis application to... Train an autoencoder network model based on a standard activity set; Extracting repetitive activities from sensor data; The trained autoencoder network model is used to generate a first score, which indicates the distance between the repetition of the repetitive activity and the standard activity. The repetitive activities are used to retrain the autoencoder network model, using the weights of the autoencoder network model as initial values, the weights being based on the autoencoder network model trained using the standard activity set; A second score is generated using the retrained autoencoder network model, indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and The final score is generated based on the weights of the first score and the second score.

9. The system according to claim 8, wherein, The processor is also configured to execute the motion analysis application to: Each of the multiple sensor inputs of the sensor data is applied to a corresponding one of the multiple encoder subnetworks of the encoder in the autoencoder network model; The outputs of the multiple encoder subnetworks are applied to another encoder subnetwork of the encoder to fuse the multiple sensor inputs to generate the extracted motion window output; and The extracted motion window output is applied to the decoder of the autoencoder network pattern, the decoder including a set of decoder subnetworks that mirror the plurality of encoder subnetworks, and the decoder regenerates the estimated motion window to minimize the reconstruction error separated to the plurality of sensor inputs.

10. The system according to claim 9, wherein, The processor is also configured to execute the motion analysis application to learn the parameters of the encoder subnetwork and the decoder subnetwork via backpropagation by minimizing the mean squared error loss.

11. The system according to claim 9, wherein, The subnetworks include one or more of fully connected networks, convolutional neural networks, long short-term memory networks, or attention networks.

12. The system according to claim 8, wherein, The processor is also configured to execute the motion analysis application to determine the weights of the first score and the second score based on one or more of the confidence level in the distance of the repetitive activity from the standard activity or the probability of sensor drift of the sensor that generated the sensor data.

13. The system of claim 8 further includes a wearable glove motion capture device, wherein the sensor data is motion capture data captured from the wearable glove motion capture device.

14. A non-transitory computer-readable medium comprising instructions for a motion analysis application, wherein when one or more processors execute the instructions for the motion analysis application, the instructions for the motion analysis application cause the one or more processors to: Train an autoencoder network model based on a standard activity set; Extracting repetitive activities from sensor data; The trained autoencoder network model is used to generate a first score, which indicates the distance between the repetition of the repetitive activity and the standard activity. The repetitive activities are used to retrain the autoencoder network model, using the weights of the autoencoder network model as initial values, the weights being based on the autoencoder network model trained using the standard activity set; A second score is generated using the retrained autoencoder network model, indicating whether the repetition is an outlier compared to other repetitions of the repetitive activity; and The final score is generated based on the weights of the first score and the second score.

15. The medium of claim 14, further comprising instructions for the motion analysis application, wherein when the one or more processors execute the instructions for the motion analysis application, the instructions for the motion analysis application cause the one or more processors to: Each of the multiple sensor inputs of the sensor data is applied to a corresponding one of the multiple encoder subnetworks of the encoder in the autoencoder network model; The outputs of the multiple encoder subnetworks are applied to another encoder subnetwork of the encoder to fuse the multiple sensor inputs to generate the extracted motion window output; and The extracted motion window output is applied to the decoder of the autoencoder network pattern, the decoder including a set of decoder subnetworks that mirror the plurality of encoder subnetworks, and the decoder regenerates the estimated motion window to minimize the reconstruction error separated to the plurality of sensor inputs.

16. The medium of claim 15, further comprising instructions for the motion analysis application, wherein when the one or more processors execute the instructions for the motion analysis application, the instructions for the motion analysis application cause the one or more processors to learn the parameters of the encoder subnetwork and the decoder subnetwork by backpropagation through minimizing the mean squared error loss.

17. The medium according to claim 15, wherein, The subnetworks include one or more of fully connected networks, convolutional neural networks, long short-term memory networks, or attention networks.

18. The medium of claim 14, further comprising instructions for the motion analysis application, wherein, when the one or more processors execute the instructions for the motion analysis application, the instructions for the motion analysis application cause the one or more processors to determine the weights of the first score and the second score based on one or more of the confidence level in the distance of the repetitive activity from the standard activity or the probability of sensor drift of the sensor that generates the sensor data.

19. The medium according to claim 14, wherein, The sensor data is motion capture data captured from a wearable glove motion capture device.

Images