Learning device, analysis device, learning method, program, and storage medium

Abstract

To provide a learning device, an analysis device, a learning method, a program, and a storage medium that can reduce data necessary for starting an analysis.SOLUTION: A learning device according to an embodiment allows learning of a first model for estimating the period of work from time series data indicating the movement of a person. The learning device acquires first time series data to which a label indicating the work is applied. The learning device extracts a pattern to be a characteristic from the period indicated by the label in the first time series data. The learning device generates timing data indicating the timing of appearance of the pattern in the first time series data. The learning device allows learning of the first model by using the label, the first time series data, and the timing data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a learning device, an analysis device, a learning method, a program, and a storage medium. [Background technology]

[0002] There is a technology that aims to improve work efficiency by analyzing human movements. It would be desirable to be able to start analyzing with as little advance preparation as possible. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-027324 A Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a learning device, an analysis device, a learning method, a program, and a storage medium that are capable of reducing the amount of data required to start an analysis. [Means for solving the problem]

[0005] A learning device according to an embodiment trains a first model that estimates the duration of an operation from time series data indicating human motion. The learning device acquires first time series data to which a label indicating the operation is assigned. The learning device extracts a characteristic pattern from the period indicated by the label in the first time series data. The learning device generates timing data indicating the appearance timing of the pattern in the first time series data. The learning device trains the first model using the label, the first time series data, and the timing data. [Brief description of the drawings]

[0006] [Figure 1]FIG. 1 is a schematic diagram illustrating a learning system according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating the processing of the learning device according to the first embodiment. [Diagram 3] 1 is an example of a working table. [Figure 4] FIG. 2 is a schematic diagram illustrating the processing of the learning device according to the first embodiment. [Diagram 5] FIG. 2 is a schematic diagram illustrating the processing of the learning device according to the first embodiment. [Figure 6] FIG. 2 is a schematic diagram illustrating the processing of the learning device according to the first embodiment. [Figure 7] FIG. 2 is a schematic diagram illustrating the processing of the learning device according to the first embodiment. [Figure 8] FIG. 2 is a schematic diagram illustrating the processing of the learning device according to the first embodiment. [Figure 9] FIG. 2 is a schematic diagram showing a neural network included in a first model. [Figure 10] FIG. 2 is a schematic diagram showing sequence data generated from time-series data. [Figure 11] FIG. 2 is a schematic diagram showing sequence data generated from time-series data. [Figure 12] FIG. 2 is a schematic diagram showing sequence data generated from time-series data. [Figure 13] 4 is a flowchart showing a learning method by the learning system according to the first embodiment. [Figure 14] (a) is an example of an image, and (b) is an example of a posture detection result. [Figure 15] FIG. 13 is a schematic diagram showing a work situation. [Figure 16] 13 is a flowchart showing a learning method by a learning system according to a modified example of the first embodiment. [Figure 17] FIG. 11 is a schematic diagram illustrating an analysis system according to a second embodiment. [Figure 18] 13 is an example of output by the analysis system according to the second embodiment. [Figure 19] 13 is an example of output by the analysis system according to the second embodiment. [Figure 20] 10 is a flowchart showing an analysis method performed by the analysis system according to the second embodiment. [Figure 21] FIG. 2 is a schematic diagram illustrating a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In this specification and the drawings, elements similar to those already described are designated by the same reference numerals, and detailed description thereof will be omitted as appropriate.

[0008] FIG. 1 is a schematic diagram illustrating a learning system according to the first embodiment. The learning system 1 according to the first embodiment includes a learning device 10, an input device 11, a display device 12, a storage device 13, and a sensor 20.

[0009] The sensor 20 collects data indicative of the motion of a person. The person is a worker performing a task. For example, the sensor 20 is an acceleration sensor or an angular velocity sensor. The sensor 20 is attached to a part of the person's body and continuously collects data on the acceleration or angular velocity of the part of the body. This provides time series data representing changes in acceleration or angular velocity over time.

[0010] The learning device 10 learns a first model. The first model estimates a period during which a worker performs a task from time-series data showing human movements. The input device 11 is used by a user to input data to the learning device 10. The display device 12 displays data output from the learning device 10. The storage device 13 stores data related to learning.

[0011] Fig. 2 and Fig. 4 to Fig. 8 are schematic diagrams for explaining the processing of the learning device according to the first embodiment. Fig. 3 is an example of a working table. The learning of the first model will be specifically described with reference to FIGS.

[0012] FIG. 2(a) is an example of time series data of acceleration acquired by the sensor 20. The horizontal axis represents time. The vertical axis represents the acceleration value. The user assigns a work label to the time series data shown in FIG. 2(a). Specifically, the label indicates the type of work, the time when the work started, and the time when the work ended. The user inputs the label to the learning device 10 using the input device 11. The label indicates the period of the work in the time series data, as shown in FIG. 2(b). In FIG. 2(b), as an example, a label L indicating that the first work is being performed is assigned. The learning device 10 accepts the input of the label and stores it in the storage device 13.

[0013] The learning device 10 associates the label input by the user with the target time series data and stores the result in the storage device 13. Here, the learning time series data to which the label has been assigned is referred to as "first time series data."

[0014] A user registers in advance the tasks that a worker can perform, separate from the labels. The registered data is stored in the storage device 13. For example, as shown in FIG. 3, a plurality of tasks are registered in a task table. When labeling, the user inputs identification information for identifying the task indicated by the label. The identification information is, for example, an ID or a name registered in the task table. For label L, an ID or a name indicating that it is the first task is input.

[0015] The learning device 10 accesses the storage device 13 and acquires the first time-series data and the label for learning the first model. For example, the learning device 10 checks the label L assigned to the first time-series data shown in FIG. 2(b). The learning device 10 refers to the working table stored in the storage device 13 and determines whether the first task indicated by the label L exists in the working table. If the first task exists in the working table, the learning device 10 associates the data obtained in the subsequent processes with the first task and appropriately stores the data in the storage device 13.

[0016] After obtaining the first time-series data and the labels, the learning device 10 may perform one or two selected from dimensional reduction and symbolization on the time-series data in order to reduce the amount of calculation.

[0017] For example, the first time series data is multi-dimensional data. When the sensor 20 collects acceleration data on three axes, the first time series data is three-dimensional data. When multiple sensors 20 are used, the first time series data is six or more dimensional data. The learning device 10 reduces the dimension of the first time series data. For example, the learning device 10 reduces the dimension by principal component analysis (PCA) to generate one-dimensional first time series data.

[0018] The learning device 10 may symbolize the values ​​of the time series data at each time. FIG. 4(a) shows the time series data collected by the sensor 20. FIG. 4(b) shows the downsampled time series data. Downsampling results in time series data whose values ​​change stepwise over time. As shown in FIG. 4(c), the learning device 10 classifies the values ​​of the time series data at each time using a plurality of thresholds b1 to b3. Depending on the classification result, the values ​​of the time series data at each time are symbolized. As a result, the character string shown in FIG. 4(d) is obtained. The character string obtained by symbolization may be used as the time series data.

[0019] The learning device 10 extracts a characteristic pattern from the period indicated by the label in the first time-series data. The pattern is a characteristic type of data that appears in the first time-series data during the execution of the task.

[0020] Fig. 5(a) is a graph that shows a schematic representation of the first time series data contracted to one dimension. In Fig. 5(a), the horizontal axis is time and the vertical axis is value. Fig. 5(b) is an example of a candidate pattern extracted from the period of label L. A candidate pattern is data that is a candidate for a pattern to be ultimately used.

[0021] For example, the user sets the length (period) of the candidate pattern in advance according to the sampling rate of the sensor 20, the agility of the movement during work, etc. In order to improve the accuracy of the analysis, it is preferable that the length of the candidate pattern is sufficiently longer than the sampling rate. Also, it is preferable to set the length of the candidate pattern shorter as the movement is more agile.

[0022] The learning device 10 may set the length of the candidate pattern to be extracted based on a length preset by a user. The learning device 10 varies the length of the candidate pattern to be extracted around the preset length, and extracts a plurality of candidate patterns of different lengths. Alternatively, the learning device 10 may set the length of the candidate pattern according to the period of the label. The learning device 10 sets a predetermined percentage of the period of the label as the length of the candidate pattern.

[0023] The learning device 10 randomly extracts a candidate pattern from the label period. The learning device 10 also extracts comparison data from the first time series data. For example, the start time of the extraction of the first comparison data is set to the start time of the first time series data. The length of the comparison data is the same as the length of the candidate pattern.

[0024] The learning device 10 calculates the similarity between the candidate pattern and the comparison data. The similarity can be a distance. The distance can be a Dynamic Time Warping (DTW) distance. If the first time series data is symbolized, the distance can be a Levenshtein distance. The learning device 10 determines the similarity based on the inverse of the distance. The obtained similarity is used as the similarity between the candidate pattern and the first time series data at the start time of the comparison data.

[0025] The learning device 10 extracts a plurality of pieces of comparison data while shifting the start time. The shift amount of the start time is set in advance. Alternatively, the shift amount of the start time may be set based on the sampling rate of the sensor 20, the length of the candidate pattern, or the period of the label. For example, when the sampling rate of the sensor 20 is 60 Hz, the learning device 10 extracts the comparison data while shifting the start time by 1 / 60 seconds.

[0026] The learning device 10 calculates the similarity between the candidate pattern and each of the multiple comparison data. The multiple similarities are used as the similarity between the candidate pattern and the first time series data at the start time of each of the multiple comparison data. In this way, the time series data of the similarity is obtained. Hereinafter, the time series data of the similarity is called "similarity data". FIG. 5(c) shows the similarity data between the first time series data shown in FIG. 5(a) and the candidate pattern shown in FIG. 5(b). In this example, the higher the similarity, the more similar the shapes of the data between the first time series data and the candidate pattern are to each other at that time.

[0027] Next, the learning device 10 randomly extracts another candidate pattern from the period of label L. The learning device 10 calculates the similarity between the other candidate pattern and each of the multiple comparison data in the same manner as the above-mentioned process. This allows similarity data regarding the other candidate pattern to be obtained. FIG. 6(a) shows the same time series data as FIG. 5(a). FIG. 6(b) shows another candidate pattern. FIG. 6(c) shows similarity data regarding the candidate pattern shown in FIG. 6(b).

[0028] The learning device 10 repeats the extraction of candidate patterns and the generation of similarity data within the period of label L until the number of candidate patterns reaches a specified number.

[0029] The learning device 10 calculates a score for each candidate pattern. The score indicates how appropriate the candidate pattern is as a pattern. The learning device 10 calculates the score of the candidate pattern based on the respective similarity data. For example, the learning device 10 counts the number of correct answers and incorrect answers of the candidate pattern. The number of correct answers is the number of times in the similarity data when the similarity exceeds a preset threshold value that occurs within the label period. The number of incorrect answers is the number of times in the similarity data when the similarity exceeds the threshold value that occurs outside the label period. The learning device 10 increases the score as the number of correct answers increases. The higher the score, the more appropriate the candidate pattern is as a pattern.

[0030] FIG. 7(a) is the similarity data shown in FIG. 5(c). In this similarity data, there are three times when the similarity exceeds the threshold value TH1. All of the times are within the period of label L. The ratio of the number of correct answers to the number of incorrect answers is 1:0. The learning device 10 sets "1" as the score of the candidate pattern shown in FIG. 5(b). FIG. 7(b) is the similarity data shown in FIG. 6(c). In this similarity data, there are nine times when the similarity exceeds the threshold value TH1. Three of the times are within the period of label L. Six of the times are outside the period of label L. The ratio of the number of correct answers to the number of incorrect answers is 0.33:0.67. The learning device 10 sets "0.33" as the score of the candidate pattern shown in FIG. 6(b). By comparing the scores, the candidate pattern shown in FIG. 5(b) is determined to be more appropriate as a pattern than the candidate pattern shown in FIG. 6(b).

[0031] The learning device 10 selects at least one of the multiple candidate patterns as a pattern based on the score. If the higher the score of a candidate pattern, the more appropriate it is as a pattern, the learning device 10 selects one or more candidate patterns in descending order of score. The learning device 10 generates timing data using the selected pattern. The timing data indicates the timing (time) at which the pattern appeared in the first time-series data.

[0032] For example, the candidate pattern shown in FIG. 5(b) is selected as a pattern. FIG. 8(a) shows the similarity data shown in FIG. 5(c). The learning device 10 identifies the time when the similarity exceeds a predetermined threshold TH2 in the similarity data. The threshold TH2 may be the same as the threshold TH1. It is considered that the pattern appears at the time when the similarity exceeds the threshold TH2. The learning device 10 assigns a value greater than 0 to the time when the similarity exceeds the predetermined threshold TH2, and assigns 0 to other times. As a result, as shown in FIG. 8(b), timing data is generated that indicates the timing at which the pattern in FIG. 5(b) appears. When multiple patterns are selected, timing data is generated for each pattern.

[0033] The learning device 10 uses the label, the first time-series data, and the timing data to train a first model. The first model estimates the duration of an activity from the time-series data indicating a person's motion.

[0034] The first model preferably includes a neural network. By using a neural network, the duration of the task can be estimated with higher accuracy. The configuration of the neural network is arbitrary. Here, an example of a neural network suitable for the first embodiment will be described.

[0035] FIG. 9 is a schematic diagram showing a neural network included in the first model. The learning device 10 arranges each value included in the time series data in a two-dimensional array of M rows x N columns to generate array data. The learning device 10 may overlay the two-dimensional data of M rows x N columns as C feature maps to generate three-dimensional array data of M rows x N columns x C maps. For example, the learning device 10 arranges each value of the time series data in chronological order as data of M rows x N columns x C maps, and inputs the data to the neural network 90. ​​Here, an example will be described in which the learning device 10 arranges each value of one-dimensional time series data as data of 1 row x N columns x 1 layer.

[0036] The input array data is input to a convolutional layer 91a and an attention layer 92a in a block 90a. The convolutional layer 91a outputs the array data of the F map from the array data of the C map. The attention layer 92a outputs the attention of the A map in addition to the array data of the F map. Attention is a distribution of weights for each vector, and indicates the area that should be emphasized in the input array data.

[0037] The array data of F map + A map is input to the maximum pooling layer 93a, and array data of 1 row × (N / 2) columns × F map is output. The output array data is input to the convolution layer 91b and attention layer 92b of the block 90b, similar to the block 90a. Array data of 1 row × (N / 2) columns × (2F map + A map) is output from the block 90b. The array data output from the block 90b is input to the convolution layer 91c and attention layer 92c of the block 90c through the maximum pooling layer 93b, and array data of 1 row × (N / 4) columns × (4F map + A map) is output. The array data output from the block 90c is input to the convolution layer 91d and attention layer 92d of the block 90d through the maximum pooling layer 93c, and array data of 1 row × (N / 8) columns × (8F map + A map) is output.

[0038] The array data output from block 90d is input to an upsampling layer 94c, where it is upsampled to array data of 1 row × (N / 4) columns × 4F map. The array data output from block 90c is transferred and concatenated to block 90e. The array data from block 90c and the array data from block 90d are superimposed and input to block 90e. In block 90e, the array data is input to a convolution layer 91e and an attention layer 92e, where array data of 1 row × (N / 4) columns × (4F map + A map) is output.

[0039] Similarly to block 90e, block 90f receives array data from block 90e through upsampling layer 94b and array data from block 90b. The array data is input to convolutional layer 91f and attention layer 92f, and array data of 1 row × (N / 2) columns × (2F map + A map) is output. Block 90g receives array data from block 90f through upsampling layer 94a and array data from block 90a. The array data is input to convolutional layer 91g, attention layer 92g, and softmax layer 95, and array data of 1 row × N columns × NL map is output.

[0040] The neural network 90 outputs an estimated result of an operation for the input sequence data. For example, when the output sequence data is in a first state, the operation corresponding to the input sequence data indicates that the labeled operation is being performed. When the output sequence data is in a second state, the operation corresponding to the input sequence data indicates that an operation other than the labeled operation is being performed.

[0041] The learning device 10 uses the first time series data as input data and the label as teaching data to train the neural network 90. ​​When the time of the input sequence data is within the period of the label, sequence data indicating that work is in progress may be added to the teaching data. When the time of the input sequence data is outside the period of the label, sequence data indicating that something other than work is being performed may be added to the teaching data. The evaluation function during learning includes a term that minimizes the difference between the data on the timing of the appearance of the pattern and the value indicating the appearance of the pattern that appears in the attention layer. Each value included in the timing data is arranged in M ​​rows x N columns x C map layers as described above.

[0042] The pattern appears frequently during the labeled task. The timing data indicates the time at which the pattern appears. In other words, the timing data indicates an area in the sequence data that should be emphasized. By using the timing data to evaluate the first model of the learning, the first model can be learned efficiently. The learning device 10 stores the learned first model in the storage device 13.

[0043] In the above, an example was described in which the first model includes a so-called U-Net structure. The first model may also include other structures (e.g., Seg-Net) that have a convolutional layer and an attention layer. The convolutional layer allows the first model to effectively extract features from time-series data. The attention layer indicates areas that should be emphasized, thereby improving the accuracy of estimation by the first model.

[0044] 10 to 12 are schematic diagrams showing sequence data generated from time-series data. The learning device 10 generates array data AD1 by arranging M-dimensional time series data in M ​​columns and N rows, for example, as shown in FIG. 10. In the example of FIG. 10, data (values) of each dimension are arranged in chronological order in the row direction. Data from the first dimension to the Mth dimension are arranged in the column direction. For example, the value V1 t1 represents the value of the first dimension of the time series data at time t1. tN represents the M-th dimension value of the time series data at time tN. The size of N is arbitrary. For example, N is set in advance by the user.

[0045] In the array data AD2 shown in Fig. 11, M-dimensional time series data is arranged in 2M columns. Data from 1st to M-dimensional times t1 to tN is arranged on the upper side in the column direction. Data from 1st to M-dimensional times t(N+1) to t2N is arranged on the lower side in the column direction. In this way, the number of data arranged in one direction can be set regardless of the number of dimensions of the time series data.

[0046] In the array data AD3 shown in Fig. 12, M-dimensional time series data is arranged in M ​​columns x N rows and overlaid on a C map. In the first map, data from 1st to M-dimensional times t1 to tN is arranged. In the second map, data from 1st to M-dimensional times t(N+1) to t2N is arranged. In the C map, data from 1st to M-dimensional times t{(C-1)N+1} to tCN is arranged. In this way, data may be overlaid in chronological order to generate a tensor of M columns x N rows x C map.

[0047] As for the timing data, the learning device 10 arranges the data two-dimensionally using the arrangement method described above, in the same manner as for the time-series data.

[0048] FIG. 13 is a flowchart showing a learning method by the learning system according to the first embodiment. The sensor 20 collects data indicating human movements (step S1). For example, the data is acceleration data, and the sensor 20 collects time-series data. The user assigns a label indicating the period of the task to the data to be used for learning (step S2). The learning device 10 acquires the label and first time-series data for learning (step S3). The learning device 10 extracts candidate patterns from the period indicated by the label in the first time-series data (step S4). The learning device 10 generates similarity data using the candidate patterns (step S5).

[0049] The learning device 10 determines whether the number of extracted candidate patterns reaches a specified number (step S6). If the number of candidate patterns is less than the specified number, the learning device 10 executes step S4 again. If the number of candidate patterns is equal to or greater than the specified number, the learning device 10 selects a pattern from the multiple candidate patterns (step S7). The learning device 10 generates timing data for the selected pattern (step S8). The learning device 10 trains a first model using the label, the first time-series data, and the timing data (step S9). The learning device 10 stores the trained first model in the storage device 13 (step S10).

[0050] The advantages of the first embodiment will be described. There is a method of extracting a pattern of each task from time series data based on a preset task sequence and task time, and estimating the duration of the task in the time series data using the pattern. This method is useful when the task sequence is fixed and the standard required time is clear. On the other hand, there are tasks whose sequence and time are not fixed. For example, at a logistics site, picking, relabeling, box assembly, packing, closing boxes, labeling, reading labels, placing on a cart, etc. are performed. The sequence and required time of these tasks may change depending on the contents of the package, the number of items to be packed, the size of each item, customer requests, etc. In addition, during one task, another task may be interrupted. There is a demand for a technology that can analyze time series data even when the timing of the execution of the task and the required time of the task are not fixed.

[0051] One such technique is to prepare a model that estimates the duration of an operation in time-series data in response to input of the time-series data. The model is trained using supervised data. However, in general, in order to obtain practical accuracy in supervised learning of a model, it is necessary to prepare a large amount of training data. Users need to spend a lot of time preparing the training data.

[0052] Regarding this problem, the learning device 10 according to the first embodiment extracts a pattern from the first time-series data to which a label is added. Then, the learning device 10 generates timing data using the pattern, and trains a first model using the label, the first time-series data, and the timing data. By using the timing data, it is possible to teach the first model the timing that should be emphasized for estimating an operation in the time-series data. This makes it possible to sufficiently improve the accuracy of the first model even with a small amount of learning data. Compared to a case where the pattern and timing data are not used, it is possible to reduce the amount of learning data required for training the first model, thereby reducing the burden on the user.

[0053] Furthermore, according to the first embodiment, the user only needs to input the start time, end time, and type of the task in labeling. After labeling, the learning device 10 automatically extracts patterns and trains the first model. Therefore, even a user who does not have specialized knowledge about labeling can easily train the first model.

[0054] In the above, an example has been described in which the first model estimates the duration of one type of task in time-series data. The first model may be trained so as to estimate the duration of multiple types of tasks in time-series data. In this case, the learning device 10 extracts patterns, generates timing data, and trains the first model for each label of the multiple types of tasks.

[0055] Specifically, the user assigns a first label to the first task and a second label to the second task to the time series data. The first label indicates that the performed task is the first task, the time when the first task started, and the time when the first task ended. The second label indicates that the performed task is the second task, the time when the second task started, and the time when the second task ended. The learning device 10 acquires first time series data to which the first label and the second label are assigned. The learning device 10 extracts a first pattern from the period indicated by the first label in the first time series data. The learning device 10 extracts a second pattern from the period indicated by the second label in the first time series data.

[0056] The learning device 10 generates first timing data indicating a timing at which a first pattern appears in the first time series data. The learning device 10 generates second timing data indicating a timing at which a second pattern appears in the first time series data. The learning device 10 trains an attention layer using a first label and the first timing data. The learning device 10 trains an attention layer using a second label and the second timing data. The learning device 10 trains a first model using the first label, the second label, and the first time series data.

[0057] (Modification) The learning device 10 may learn the first model by focusing on only some dimensional time series data included in the multidimensional time series data. For example, the sensor 20 is an image sensor included in an imaging device. The sensor 20 continuously collects images showing human movements. The imaging device stores the obtained video in the storage device 13.

[0058] The learning device 10 detects the posture of a person in an image. The posture is represented by the positions of the person's parts and joints. OpenPose, etc., can be used to detect the posture. A part refers to a part of the body, such as the eyes, ears, nose, head, shoulders, upper arms, forearms, hands, chest, abdomen, thighs, lower legs, and feet. A joint refers to a movable joint that connects at least some of the parts, such as the neck, elbows, wrists, waists, knees, and ankles.

[0059] Fig. 14(a) is an example of an image, and Fig. 14(b) is an example of the posture detection result for Fig. 14(a). The learning device 10 accesses the storage device 13 and acquires a video. The learning device 10 acquires a still image of a person 96 from the video, as shown in FIG. 14(a). The learning device 10 detects the posture of the person in the image. For example, as shown in FIG. 14(b), the positions of the head 97a, left shoulder 97b, left elbow 97c, left wrist 97d, left hand 97e, right shoulder 97f, right elbow 97g, right wrist 97h, right hand 97i, and spine 97j of the person 96 are detected.

[0060] The learning device 10 sequentially performs posture detection on the images at each time of the video. This allows obtaining time-series data indicating the position of each part and each joint at each time. In the example shown in FIG. 14(b), the positions in the X direction and the Y direction are collected as two-dimensional data for each of the head 97a, left shoulder 97b, left elbow 97c, left wrist 97d, left hand 97e, right shoulder 97f, right elbow 97g, right wrist 97h, right hand 97i, and spine 97j, and 20-dimensional time-series data is obtained as a whole.

[0061] Study device 10 displays a video captured by an imaging device on display device 12. A user inputs a label to study device 10 using input device 11. Specifically, the user inputs the start time and end time of a task and the identification information of that task while checking the displayed video.

[0062] The learning device 10 extracts m-dimensional time series data with large fluctuations during a period indicated by a label from the entire n-dimensional time series data. m and n are integers. m is smaller than n. For example, the learning device 10 calculates the average value and the degree of dispersion of the entire time series data during the period of the label. The learning device 10 extracts a part of the time series data with a large degree of dispersion. For example, the variance or standard deviation is used as the degree of dispersion.

[0063] The learning device 10 extracts a pattern from the period indicated by the label in the extracted first time series data. The pattern is extracted from the time series data of each dimension. That is, an m-dimensional pattern is extracted from m-dimensional time series data. The learning device 10 generates m pieces of similarity data and m pieces of timing data using the m-dimensional pattern and the m-dimensional time series data. The learning device 10 may generate m-dimensional similarity data and m-dimensional timing data. Thereafter, the first model is learned using the first time series data and the timing data in the same manner as the above-mentioned process.

[0064] 15(a) and 15(b) are schematic diagrams showing the state of work. In this example, extracting a portion of the time series data with large fluctuations corresponds to focusing on a portion of the body parts with large fluctuations. For example, in FIG. 15(a), a person attaches a label to a package. Then, as shown in FIG. 15(b), the person picks up a barcode reader placed next to him and reads the barcode on the attached label. At this time, the positions of the right hand, right wrist, right forearm, and right elbow in particular change significantly. Extracting a portion of the time series data by the learning device 10 corresponds to extracting the changes in the positions of the right hand, right wrist, right forearm, and right elbow from the changes in the positions of each body part and each joint.

[0065] FIG. 16 is a flowchart showing a learning method by a learning system according to a modified example of the first embodiment. After step S3, the learning device 10 extracts time series data of some dimensions from the multidimensional first time series data (step S12). After that, the learning device 10 executes step S4 and subsequent steps on the first time series data from which some dimensions have been extracted, in the same manner as in the flowchart shown in FIG.

[0066] According to this modification, it is possible to more accurately estimate an activity that includes a characteristic motion, such as a large movement of a specific part of the body in a specific direction.

[0067] When arranging time series data acquired from a person's posture to input it to the first model, it is preferable that data of adjacent parts or joints are adjacent to each other in the arrangement data. For example, data of the right hand, right ankle, right forearm, right elbow, and right upper arm are extracted as data with large fluctuations. The extracted first time series data is 10-dimensional data representing the positions of these parts and joints in the X and Y directions. When arranging data as shown in FIG. 10, it is preferable that adjacent parts and joints are adjacent to each other in the arrangement data as follows. Values ​​V1 and V2 are the positions of the right hand in the X and Y directions, respectively. Values ​​V3 and V4 are the positions of the right wrist in the X and Y directions, respectively. Values ​​V5 and V6 are the positions of the right forearm in the X and Y directions, respectively. Values ​​V7 and V8 are the positions of the right elbow in the X and Y directions, respectively. Values ​​V9 and V10 are the positions of the right upper arm in the X and Y directions, respectively. When data on adjacent parts or joints are adjacent in the sequence data, features can be more easily extracted from the sequence data, and the accuracy of estimation by the first model can be improved.

[0068] Second embodiment FIG. 17 is a schematic diagram illustrating an analysis system according to the second embodiment. Compared to the learning system 1, the analysis system 2 according to the second embodiment further includes an analysis device 30.

[0069] The sensor 20 collects data (e.g., acceleration data or images) indicating the movement of the analysis target. The analysis device 30 accesses the storage device 13 and acquires the data indicating the movement and the first model learned by the learning device 10. The analysis device 30 uses the data collected by the sensor 20 as time series data to be input to the first model. Alternatively, the analysis device 30 generates time series data based on the data collected by the sensor 20. Here, the time series data for analysis input to the first model is called "second time series data."

[0070] The analysis device 30 inputs the second time series data into the first model. The analysis device 30 obtains an estimation result by the first model as an analysis result of the second time series data. The analysis result indicates a period in the second time series data during which the task was performed. The analysis device 30 stores the analysis result in the storage device 13.

[0071] The accuracy of the analysis can be improved by using the first model learned by the learning device 10 for the analysis. For example, even for a job in which the order of tasks, the time required for each task, etc. are not fixed, the duration of the task can be analyzed with higher accuracy based on the time-series data indicating the actions.

[0072] 18 and 19 are examples of outputs from the analysis system according to the second embodiment. Here, an example will be described in which a worker performs picking, relabeling, box assembly, packing, closing, labeling, label reading, and placing on a cart at a logistics site.

[0073] The analysis device 30 displays the analysis results on the display device 12. For example, as shown in Fig. 18, the analysis device 30 displays the period of each task in the second time series data together with the second time series data and task information. In this example, the name of the task is displayed as the task information.

[0074] Alternatively, when the duration of the tasks is estimated by the first model, the analysis device 30 acquires the type of tasks performed, the start time of each task, and the end time of each task. The analysis device 30 displays the tasks performed in chronological order, as shown in FIG. 19. In this case, as shown in FIG. 19, the analysis device 30 may display the start time of a job including a group of tasks. The analysis device 30 may display tasks of the same type in the same manner, and tasks of different types in different manners.

[0075] The start of a job is determined based on one or more of the following: recognition of a worker in the image, movement of the worker to a predetermined location in the image, and the appearance of any pattern. The end of a job is determined based on one or more of the following: absence of a worker in the image, movement of the worker outside the image, movement of the worker to a predetermined location in the image, and the passage of a preset time.

[0076] In the example shown in FIG. 19, actions other than work are specified as "other" in the time series data. For example, the analysis device 30 may calculate a similarity between the second time series data and a pattern, and generate similarity data regarding the second time series data. The analysis device 30 uses this similarity data to generate timing data indicating the appearance timing of the pattern in the second time series data. The analysis device 30 determines that the work is completed if the pattern does not appear for a predetermined period of time. The analysis device 30 determines that there is an action other than work during at least a part of the period from the last appearance of the pattern to the next appearance of the pattern. The "other" action corresponds to the action other than work registered in the work table shown in FIG. 3.

[0077] The analysis device 30 may extract state change points included in the time series data. For example, a hidden Markov model for a hierarchical Deriche process (HDP-HMM), a k-means method, an x-means method, or a spectral clustering method is used to extract state change points. The analysis device 30 determines that an action other than the work is present between any change point after the last pattern appears and any other change point before the next pattern appears.

[0078] The analysis device 30 may determine a non-task behavior based on both the appearance timing of the pattern in the second time series data and the change point of the state in the second time series data. For example, the analysis device 30 determines that the task has been completed at the change point immediately after the last appearance of the pattern. The analysis device 30 determines that the task has been started at the change point immediately before the next appearance of the pattern.

[0079] By displaying the analysis results, the user can easily understand what tasks the worker is performing, in what order, and for how long.

[0080] FIG. 20 is a flowchart showing an analysis method performed by the analysis system according to the second embodiment. The sensor 20 collects data indicating the actions of a person to be analyzed (step S21). The analysis device 30 acquires a first model that has been learned by the learning device 10 (step S22). The analysis device 30 acquires second time-series data to be input to the first model (step S23). The analysis device 30 inputs the second time-series data to the first model and performs analysis (step S24). The analysis device 30 outputs the analysis result (step S25).

[0081] FIG. 21 is a schematic diagram showing a hardware configuration. For example, the learning device 10 is a computer and has a Read Only Memory (ROM) 10a, a Random Access Memory (RAM) 10b, a Central Processing Unit (CPU) 10c, and a Hard Disk Drive (HDD) 10d.

[0082] The ROM 10a stores a program for controlling the operation of the computer, ie, a program required for causing the computer to execute each of the above-mentioned processes.

[0083] The RAM 10b functions as a storage area in which the programs stored in the ROM 10a are loaded. The CPU 10c includes a processing circuit. The CPU 10c reads the control programs stored in the ROM 10a and controls the operation of the computer according to the control programs. The CPU 10c also loads various data obtained by the operation of the computer in the RAM 10b. The HDD 10d stores data necessary for the processing of the learning device 10 and data obtained by the processing. The HDD 10d may function as a storage device 13.

[0084] The learning device 10 may have an embedded Multi Media Card (eMMC), a Solid State Drive (SSD), a Solid State Hybrid Drive (SSHD), etc. instead of the HDD 10d.

[0085] The input device 11 includes, for example, at least one selected from a mouse, a keyboard, a microphone (voice input), and a touch pad. The display device 12 includes, for example, at least one selected from a monitor and a projector. A device having the functions of both the input device 11 and the display device 12, such as a touch panel, may be used.

[0086] The hardware configuration of the analysis device 30 can be the same as that of the learning device 10. One computer may function as the learning device 10 and the analysis device 30. Alternatively, the respective processes and functions of the learning device 10 and the analysis device 30 may be realized by the cooperation of more computers.

[0087] According to the learning device, learning system, or learning method described above, the accuracy of the first model can be improved with less learning data. According to the analysis device, analysis system, or analysis method, the accuracy of analysis can be improved by analyzing time-series data using the trained first model. In addition, the same effect can be obtained by using a program for operating a computer as a learning device or an analysis device.

[0088] The various data processing operations described above may be stored as a program that can be executed by a computer on a non-transitory computer readable storage medium, such as a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), or a semiconductor memory.

[0089] For example, data recorded on a storage medium can be read by a computer (or an embedded system). The storage medium may have any recording format (storage format). For example, the computer reads a program from the storage medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program through a network.

[0090] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0091] 1: learning system, 2: analysis system, 10: learning device, 11: input device, 12: display device, 13: storage device, 20: sensor, 30: analysis device, 90: neural network, 90a-90g: block, 91a-91g: convolution layer, 92a-92g: attention layer, 93a-93c: max pooling layer, 94a-94c: upsampling layer, 95: softmax layer, 96: person, 97a: head, 97b: left shoulder, 97c: left elbow, 97d: left wrist, 97e: left hand, 97f: right shoulder, 97g: right elbow, 97h: right wrist, 97i: right hand, 97j: spine, AD1-AD3: sequence data, L: label, TH1,TH2: threshold, b1-b3: threshold

Claims

1. A learning device that learns a first model that estimates a period of an operation from time-series data indicating a human movement, comprising: Obtaining first time-series data to which a label indicating the work is assigned; Extracting a characteristic pattern from a period indicated by the label in the first time series data; generating time series data of similarity by calculating similarity between each of a plurality of comparison data extracted from the first time series data and the pattern; generating timing data indicating an appearance timing of the pattern in the first time-series data by comparing the time-series data of the similarity with a preset threshold value and binarizing the data; A learning device that uses the label, the first time-series data, and the timing data to train the first model.

2. A learning device that learns a first model that estimates a period of an operation from time-series data indicating a human movement, comprising: Obtaining first time-series data to which a label indicating the work is assigned; extracting a plurality of candidate patterns and a plurality of comparison data from a period indicated by the label in the first time-series data; Calculating a similarity between each of the plurality of candidate patterns and the plurality of comparison data; selecting one pattern that is a feature of the first time series data from the plurality of candidate patterns based on the calculated similarity; generating timing data indicating appearance timings of the pattern in the first time series data; A learning device that trains the first model using the label, the first time-series data, and the timing data.

3. Reducing the dimensions of the multidimensional first time series data; The learning device according to claim 1 , wherein the pattern is extracted from the condensed first time-series data.

4. The first time-series data is multi-dimensional data representing motions of a plurality of body parts and motions of a plurality of joints, The learning device includes: Extracting the first time series data of a part of dimensions; Extracting the pattern from the extracted first time series data for each of the some dimensions and generating the timing data; training the first model using the labels, the extracted first time series data, and the plurality of timing data; The learning device according to claim 1 or 2.

5. A plurality of the labels indicating the plurality of the operations are assigned to the first time-series data, The learning device includes: extracting a plurality of the patterns from a plurality of periods indicated by the plurality of labels, generating a plurality of pieces of timing data each indicating an appearance timing of the plurality of patterns in the first time series data; The learning device according to any one of claims 1 to 4, wherein the first model is trained so as to estimate periods of the plurality of tasks from the time series data using the plurality of labels, the first time series data, and the plurality of timing data.

6. generating array data to be input to the first model by arranging each value of the first time series data in a two-dimensional form; The learning device according to claim 1 , wherein the first model is trained using the label, the sequence data based on the first time-series data, and the timing data.

7. The first model includes a neural network having a convolutional layer and an attention layer; the learning device uses the first time-series data as input data and the label as teaching data to learn the first model; 7. The learning device according to claim 1, wherein the timing data is used to evaluate the first model during learning.

8. 8. The learning device according to claim 1, wherein data indicating a type of the task, a start time of the task, and an end time of the task is accepted as the label.

9. An analysis device that inputs second time series data to be analyzed into the first model learned by the learning device described in any one of claims 1 to 8, and analyzes the period of the work in the second time series data.

10. The analysis device according to claim 9 , further comprising: an analysis unit configured to analyze the period of the non-work action based on one or both of the timing at which the pattern appears in the second time series data and the point of change in state in the second time series data.

11. A learning method for learning, using a computer, a first model that estimates a period of an operation from time-series data showing a human movement, comprising: The computer includes: acquiring first time-series data to which a label indicating the work is assigned; extracting a characteristic pattern from a period indicated by the label in the first time series data; generating time series data of similarity by calculating similarity between each of a plurality of comparison data extracted from the first time series data and the pattern; generating timing data indicating an appearance timing of the pattern in the first time-series data by comparing the time-series data of the similarity with a preset threshold and binarizing the data; training the first model using the labels, the first time series data, and the timing data; How to learn.

12. A learning method for learning, using a computer, a first model that estimates a period of an operation from time-series data showing a human movement, comprising: The computer includes: acquiring first time-series data to which a label indicating the work is assigned; extracting a plurality of candidate patterns and a plurality of comparison data from a period indicated by the label in the first time-series data; Calculating a similarity between each of the plurality of candidate patterns and the plurality of comparison data; selecting one pattern that is characteristic of the first time-series data from the plurality of candidate patterns based on the calculated similarity; generating timing data indicating appearance timings of the pattern in the first time series data; training the first model using the labels, the first time series data, and the timing data; How to learn.

13. A program for causing a computer to execute the learning method according to claim 11 or 12.

14. A storage medium storing the program according to claim 13.

Images