Labeling system, activity recognition system, and teaching material generation system

Abstract

To easily and properly label data that corresponds to an activity included in time-series data.SOLUTION: A labeling system 1 includes a processor (102, 103, 104) and a machine learning model from which, in response to an input of time series data, a high-level label indicating a high-level activity included in the time series data and low-level labels indicating a plurality of low-level activities included in the high-level activity are output. The processor is configured to output a high-level label and a low-level label by using the machine learning model and cause the high-level label and the low-level label to be displayed, accept an input regarding a high-level label included in the time series data, train the machine learning model assuming that a high-level activity corresponding to the accepted high-level label is included in the time series data, and output a high-level label and a low-level label by using the trained machine learning model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for labeling data corresponding to activities included in time-series data. [Background technology]

[0002] Activity recognition based on time series data is a major research topic with applications in a wide variety of domains. In recent years, the rapid improvement of machine learning techniques such as deep learning has opened up new application possibilities for activity recognition in industrial and commercial environments.

[0003] One of the key problems faced in deploying machine learning models for activity recognition is the need to label the large amounts of training data required to build the machine learning models.

[0004] To address this problem, US Patent No. 6,239,663 shows a method for reducing the amount of training data that needs to be labeled by actively selecting specific unlabeled data samples based on the expected improvement in model performance when labeled given unlabeled data samples.

[0005] However, this method relies on reducing the amount of training data processed to improve labeling efficiency, and does not address the case where one wants to review and label all of the unlabeled time series data they are processing.

[0006] On the other hand, Non-Patent Document 1 discloses a method for improving the efficiency of processing all unlabeled data. This method improves performance by automatically labeling each repetition in an activity cycle, and is applicable only to counting repetitions of simple, periodic activities, such as walking or running. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] US Patent Application Publication No. 2007-133664 [Non-patent literature]

[0008] [Non-Patent Document 1] Martindale, Christine F and Hoenig, Florian and Strohrmann, Christina and Eskofier, Bjoern M,”Smart annotation of cyclic data using hierarchical hidden Markov models”, Sensors, Multidisciplinary Digital Publishing Institute,2017,Vol.17,No.10,P.2328 Summary of the Invention [Problem to be solved by the invention]

[0009] The process of labeling time series data for use as training data for machine learning models can be costly, and there is a need to improve the efficiency of this process.

[0010] The technique in US Pat. No. 6,399,363 is unable to address the need to label everything, including rare events, in unlabeled data, such as when building a model that can recognize rare events in data.

[0011] On the other hand, Non-Patent Document 1 can only be applied to labeling simple repetitive activities such as walking and running, limiting its use cases.

[0012] The present invention has been made in consideration of the above circumstances, and has an object to provide a technique that can easily and appropriately label data corresponding to activities included in time-series data. [Means for solving the problem]

[0013] In order to achieve the above object, a labeling system according to one aspect is a labeling system that performs labeling on activities included in time-series data, the activities including low-level activities and high-level activities composed of a plurality of the low-level activities. The labeling system includes a processor and a machine learning model that receives time-series data as input and outputs a high-level label that is a label indicating a high-level activity included in the time-series data and a low-level label indicating a plurality of low-level activities included in the high-level activities. The processor uses the machine learning model to output high-level labels and low-level labels for the high-level activities and low-level activities included in the target time-series data, and displays the high-level labels and low-level labels for the high-level activities and low-level activities included in the outputted target time-series data. The processor receives input for the high-level labels included in the time-series data, and trains the machine learning model assuming that the time-series data includes a high-level activity corresponding to the received high-level label. The labeling system uses the trained machine learning model to output high-level labels and low-level labels for the high-level activities and low-level activities included in the target time-series data. Effect of the Invention

[0014] According to the present invention, data corresponding to activities contained in time-series data can be easily and appropriately labeled. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a functional configuration diagram of a labeling system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining the processing operation of the hierarchical activity recognition model training unit according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart of a hierarchical activity recognition model training process according to the first embodiment. [Figure 4]FIG. 4 is a configuration diagram of a time-series data set according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing the configuration of a low-level class probability dataset according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing the configuration of a low-level label data set according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing the configuration of a high-level label data set according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating the processing of the machine annotation unit according to the first embodiment. [Figure 9] FIG. 9 is a flowchart of the machine annotation process according to the first embodiment. [Figure 10] FIG. 10 is a flowchart of the overall process according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating a user interface according to the first embodiment. [Figure 12] FIG. 12 is a configuration diagram of a computer that constitutes the labeling system according to the first embodiment. [Figure 13] FIG. 13 is a functional configuration diagram of a labeling system according to the second embodiment. [Figure 14] FIG. 14 is a diagram for explaining the processing of the machine annotation / part reconstruction unit according to the second embodiment. [Figure 15] FIG. 15 is a flowchart illustrating a machine annotation / partial reconstruction process according to the second embodiment. [Figure 16] FIG. 16 is a flowchart of the overall process according to the second embodiment. [Figure 17] FIG. 17 is a diagram illustrating the processing of the machine annotation unit according to the third embodiment. [Figure 18] FIG. 18 is a flowchart of the machine annotation process according to the third embodiment. [Figure 19] FIG. 19 is a diagram showing the configuration of a high-level label priority data set according to the third embodiment. [Figure 20]FIG. 20 is a diagram showing the configuration of a low-level label priority data set according to the third embodiment. [Figure 21] FIG. 21 is a functional configuration diagram of an activity recognition system according to the fourth embodiment. [Figure 22] FIG. 22 is a flowchart of the error detection process according to the fourth embodiment. [Diagram 23] FIG. 23 is a configuration diagram of an error detection probability data set according to the fourth embodiment. [Figure 24] FIG. 24 is a functional configuration diagram of the teaching material generating system according to the fifth embodiment. [Diagram 25] FIG. 25 is a flowchart of the teaching material generation process according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The following embodiments will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the invention, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention.

[0017] In the following description, information may be described using the expression "AAA dataset", but the information may be expressed in any data structure. In other words, to indicate that the information is independent of the data structure, the "AAA dataset" may be called "AAA information".

[0018] FIG. 1 is a functional configuration diagram of a labeling system according to the first embodiment.

[0019] The labeling system 1 stores a labeled time-series data dataset 101 and an unlabeled time-series data dataset 105 , and includes a hierarchical activity recognition model training unit 102 , a machine annotation unit 103 , and a human annotation unit 104 .

[0020] The labeled time-series data set 101 stores labeled time-series data. Here, the time-series data is, for example, video data of a person performing an activity or time-series sensor data obtained by a sensor attached to the person performing an activity. An activity is, for example, an activity (high-level activity) in which a person creates a product (for example, a washing machine, a refrigerator, etc.), and each activity (high-level activity) of creating a product includes a plurality of lower activities (low-level activities) included in the activity of creating the product. For example, in the case of creating a washing machine, low-level activities include an activity of attaching a panel, an activity of inserting a screw, an activity of inserting a bolt, etc. Labels for the time-series data include high-level labels indicating high-level activities in the time-series data and low-level labels indicating low-level activities. For the time-series data in the labeled time-series data set 101, a high-level label is attached to a period in which a high-level activity in the time-series data is performed (high-level activity period), and a low-level label is attached to a partial period (subsegment) in the high-level activity period in which a low-level activity is performed. In addition, labeled time series data may be, for example, data in which a high-level activity in the time series data for a certain activity is labeled with a high-level label for the high-level activity and a low-level label for the low-level activity included therein.

[0021] The hierarchical activity recognition model training unit 102 trains a machine learning model (hierarchical activity recognition model) for estimating high-level labels and low-level labels in unlabeled time-series data, using labeled time-series data.

[0022] The unlabeled time series data set 105 stores unlabeled time series data, that is, time series data to be labeled by the labeling system 1.

[0023] The machine annotation unit 103 inputs the unlabeled time series data of the unlabeled time series data set 105 into the machine learning model trained by the hierarchical activity recognition model training unit 102 to estimate high-level and low-level labels to be attached to the time series data.

[0024] The human annotation unit 104 displays the time series data, together with the high-level and low-level labels to be attached to the time series data estimated by the machine annotation unit 103, in a user interface (see, for example, FIG. 11 ), and accepts input for correcting the estimated labels (high-level and / or low-level labels) and creating new labels (high-level and / or low-level labels) for the time series data. In this case, the human annotation unit 104 converts the unlabeled time series data into labeled time series data based on the input, and adds it to the labeled time series data dataset 101.

[0025] Thereafter, the hierarchical activity recognition model training unit 102 may use the labeled time-series data set 101 to train a machine learning model, and the machine annotation unit 103 may perform similar processing as described above using the newly trained machine learning model.

[0026] Next, the hierarchical activity recognition model training process 300 of the hierarchical activity recognition model training unit 102 will be described.

[0027] FIG. 2 is a diagram for explaining the processing operation of the hierarchical activity recognition model training unit according to the first embodiment, and FIG. 3 is a flowchart of the hierarchical activity recognition model training process according to the first embodiment.

[0028] The hierarchical activity recognition model training unit 102 inputs the time series data S of the time series data dataset 201 (see FIG. 4), the low level labels L of the low level label dataset 205 (see FIG. 6), and the high level labels H of the high level label dataset 207 (see FIG. 7) (step 301). Here, the labeled time series data dataset 101 and the unlabeled time series data dataset 105 in FIG. 1 are composed of the time series data dataset 201, the low level label dataset 205, and the high level label dataset 207.

[0029] The hierarchical activity recognition model training unit 102 constructs a low-level model 202 and trains the low-level model 202 using the time-series data and the authenticated low-level label L (an entry with Authenticated 606 set to True) (step 302). The low-level model 202 inputs the time-series data and outputs the probability (low-level class probability) of belonging to each assumed low-level activity type (low-level class) for the low-level activity included in the time-series data in a time series. The low-level model 202 may be a deep learning model, for example, a Long Short-Term memory (LSTM).

[0030] The hierarchical activity recognition model training unit 102 inputs the time series data S to the low-level model 202 and generates a low-level class probability P at each time (timestamp) for the time series data S (step 303). The generated low-level class probability P is stored in the low-level class probability dataset 203.

[0031] The hierarchical activity recognition model training unit 102 constructs a high-level model 204 and trains it using the low-level class probabilities and the authenticated high-level labels H (step 304). The high-level model 204 inputs the low-level class probabilities and outputs the high-level labels. The high-level model 204 is, for example, a KNN (K-Nearest Neighbor) classifier. Note that the high-level model 204 may be a deep learning model, for example, a LSTM (Long Short-Term memory), but using a simpler model allows training with less training data.

[0032] The hierarchical activity recognition model training unit 102 constructs a sequential model 206, and for each class (target class c) of each high-level activity type (high-level class), trains the sequential model 206 using a low-level class probability P, a high-level label H (referred to as M) that has been authenticated and whose high-label class is the target class c, and a low-level label L that has been authenticated and whose parent label is M (step 305). The sequential model 206 is a model associated with each high-level class, and inputs the low-level class probability P and outputs the low-level label L of the low-level activity in the high-level activity corresponding to the corresponding high-level class. The sequential model 206 is, for example, an HMM (Hidden Markov Model) in which the low-level class probability is an observable quantity and the low-level label is a hidden state.

[0033] The hierarchical activity recognition model training unit 102 outputs low-level class probabilities P that are used by the machine annotation unit 103 (step 306).

[0034] Next, the configuration of the time-series data set 201 will be described.

[0035] FIG. 4 is a configuration diagram of a time-series data set according to the first embodiment.

[0036] The time series data set 201 stores an entry corresponding to each sample of time series data. The entry of the time series data set 201 includes columns of a timestamp 401 and time series data 402.

[0037] The timestamp 401 stores a timestamp indicating the time of the sample corresponding to the entry. For example, if there are H samples in the time series data set 201, the timestamp of the first sample is denoted by T1, the timestamp of the last sample is denoted by T H The time series data 402 stores data at the time of the sample corresponding to the entry. The data stored in the time series data 402 may be composed of multiple objects or values. For example, in the case of data from a three-axis accelerometer, the time series data may be composed of values ​​of each of the three axes. In the example of FIG. 4, the Jth value in the data of the Hth sample is represented by S H , J As shown in Fig.

[0038] Next, the construction of the low-level class probability data set 203 will be described.

[0039] FIG. 5 is a diagram showing the configuration of a low-level class probability dataset according to the first embodiment.

[0040] The low-level class probability dataset 203 stores an entry for each time point of a given sample. An entry in the low-level class probability dataset 203 includes columns for a timestamp 501 and a low-level class probability 502.

[0041] A timestamp indicating the time of the sample corresponding to the entry is stored in the timestamp 501. For example, if there are H samples in the low-level class probability dataset 203, the timestamp of the first sample is denoted by T1, the timestamp of the last sample is denoted by T HIn this embodiment, the sample times in the low-level class probability data set 203 correspond one-to-one to the sample times in the time-series data set 201, but may correspond in other ways.

[0042] The low-level class probability 502 stores the probability of each low-level class included in the time series data at the time of the timestamp corresponding to the entry (low-level class probability: low-level activity probability). The low-level class probability is a vector of probabilities corresponding to each level class. The low-level class probability includes one probability value for each of all K classes targeted by the low-level model 202. In FIG. 5, the probability value of the Kth low-level class at the Hth timestamp is represented as P H (LC K ) are indicated.

[0043] Next, the configuration of the low-level label data set 205 will be described.

[0044] FIG. 6 is a diagram showing the configuration of a low-level label data set according to the first embodiment.

[0045] The low-level label dataset 205 stores an entry for each low-level label for time series data. An entry in the low-level label dataset 205 includes columns for a low-level label 601, a parent high-level label 602, a start time 603, an end time 604, a label class 605, and authenticated 606.

[0046] An index indicating a low-level label corresponding to an entry is stored in the low-level label 601. If there are N low-level labels in the low-level label dataset 205, the index of the first low-level label is indicated by LL1, the index of the last low-level label is indicated by LL2, NThe parent high-level label 602 stores an index (high-level label index) in the high-level label dataset 207 of the parent level label that includes the low-level label corresponding to the entry. The parent high-level label 602 in the low-level label dataset 205 stores 0 or more high-level label indexes.

[0047] The start time 603 stores the time (start time) when the low-level activity corresponding to the low-level label corresponding to the entry starts in the time series data. If there are N low-level labels in the low-level label dataset 205, the start time of the first low-level label is indicated by LS1, and the start time of the last low-level label is indicated by LS2. N The end time 604 stores the time (end time) when the low-level activity corresponding to the low-level label corresponding to the entry ends in the time series data. If there are N low-level labels in the low-level label dataset 205, the end time of the first low-level label is indicated by LE1, the end time of the last low-level label is indicated by LE2, and so on. N In this embodiment, it is assumed that two low-level labels do not overlap in the time series data, and two low-level labels LLa and LLb do not exist in the low-level label data set 205 such that the start time LSa<the end time LEb and the start time LSb<the end time LEa.

[0048] The label class 605 stores a low-level label class indicating the low-level activity of the low-level label corresponding to the entry. In this embodiment, there are K low-level label classes, which is the total number of classes targeted by the low-level model 202. The label class 605 in the low-level label dataset 205 stores 0 or more low-level label classes. The authenticated 606 stores a Boolean value, i.e., True or False, indicating whether a human annotator has confirmed the correctness of the low-level label corresponding to the entry. The value of authenticated 606 is used to determine which label is training data when training a hierarchical activity recognition model.

[0049] Next, the configuration of the high-level label data set 207 will be described.

[0050] FIG. 7 is a diagram showing the configuration of a high-level label data set according to the first embodiment.

[0051] The high-level label dataset 207 stores an entry for each high-level label for time series data. An entry of the high-level label dataset 207 includes columns of a high-level label 701, a start time 702, an end time 703, a label class 704, and authenticated 705.

[0052] An index indicating a high-level label corresponding to an entry is stored in the high-level label 701. If there are M high-level labels in the high-level label dataset 207, the index of the first high-level label is denoted by HL1, the index of the last high-level label is denoted by HL M As shown in Fig.

[0053] The start time 702 stores the time (start time) when the high-level activity corresponding to the high-level label corresponding to the entry starts in the time series data. If there are M high-level labels in the high-level label dataset 207, the start time of the first high-level label is indicated by HS1, and the start time of the last high-level label is indicated by HS2. M The end time 703 stores the time (end time) when the high-level activity corresponding to the high-level label corresponding to the entry ends in the time series data. If there are M high-level labels in the high-level label dataset 207, the end time of the first high-level label is indicated by HE1, and the end time of the last low-level label is indicated by HE2. M In this embodiment, it is assumed that two high-level labels do not overlap in the time series data, and two high-level labels HLa and HLb do not exist in the high-level label data set 207 such that the start time HSa<the end time HEb and the start time HSb<the end time HEa.

[0054] The label class 704 stores a high-level label class indicating the high-level activity of the high-level label corresponding to the entry. In this embodiment, there are J high-level label classes, which is the total number of classes targeted by the high-level model 204. The label class 704 in the high-level label dataset 207 stores 0 or more high-level label classes. The authenticated 705 stores a Boolean value indicating whether or not a human annotator has confirmed the correctness of the high-level label corresponding to the entry. The value of authenticated 705 is used to determine which labels are training data when training a hierarchical activity recognition model.

[0055] Next, the machine annotation process 900 of the machine annotation unit 103 will be described.

[0056] Fig. 8 is a diagram for explaining the processing of the machine annotation unit according to the first embodiment. Fig. 9 is a flowchart of the machine annotation processing according to the first embodiment.

[0057] The machine annotation unit 103 receives as input the low-level labels L of the low-level label dataset 205, the high-level labels H of the high-level label dataset 207, and the low-level class probabilities P of the low-level class probability dataset 203 (step 901).

[0058] The machine annotation unit 103 deletes unauthenticated labels from the high-level label H and the low-level label L (step 902). Specifically, the machine annotation unit 103 sets the labels in the high-level label H for which the value of Authenticated 705 is True as the high-level label H, and sets the labels in the low-level label L for which the value of Authenticated 606 is True as the low-level label L.

[0059] The machine annotation unit 103 inputs the low-level class probability P obtained from the time series data into the high-level model 204 to predict high-level labels R for unlabeled time intervals in the time series data, and sets authenticated to False for all predicted high-level labels R (step 903).

[0060] The machine annotation unit 103 mixes the high-level label R with the high-level label H and sets it as a new high-level label H (step 904).

[0061] For each high-level label h in the high-level labels H, the machine annotation unit 103 predicts the sequence of low-level labels N for that time interval by running the sequential model 206 corresponding to the high-level label h for the time interval corresponding to the high-level label h in the low-level class probability dataset 203. The machine annotation unit 103 then sets the authenticated columns for these predicted low-level labels N to False (step 905).

[0062] The machine annotation unit 103 mixes the low-level label N with the low-level label L to set a new low-level label L (step 906). The machine annotation unit 103 then passes the low-level label L and the high-level label H as output to the human annotation unit 104 for review in human annotation (step 907).

[0063] Next, the overall processing including the human annotation processing by the human annotation unit 104 will be described.

[0064] FIG. 10 is a flowchart of the overall process according to the first embodiment.

[0065] In the overall process, the hierarchical activity recognition model training unit 102 executes the hierarchical activity recognition model training process 300 (step 300). Then, the machine annotation unit 103 executes the machine annotation process (step 900). Then, the human annotation unit 104 executes the human annotation process 1000.

[0066] In the human annotation process 1000, the human annotation unit 104 displays the low-level labels and high-level labels output by the machine annotation process 900 and the time-series data in a user interface (e.g., a GUI (Graphical User Interface)) as shown in Fig. 11 (step 1001). This allows the user to review the set labels.

[0067] When the human annotation unit 104 receives an addition of a new label (high-level label and / or low-level label) from the user via the user interface, the human annotation unit 104 adds an entry for the label to the corresponding dataset (the high-level label dataset 207 in the case of adding a high-level label, and the low-level label dataset 205 in the case of adding a low-level label) (step 1002), sets the authenticated column of the entry to True (step 1003), and proceeds to step 1006. Through this process, the authenticated new label added by the user can be stored in the dataset (the high-level label dataset 207 and / or the low-level label dataset 205).

[0068] On the other hand, when the human annotation unit 104 receives an adjustment (change instruction) for an existing label (high-level label and / or low-level label) from the user via the user interface, the human annotation unit 104 adjusts the corresponding label entry to an adjusted value (step 1004). Next, the human annotation unit 104 determines whether or not the authenticated has been set to True by the user (step 1005), and if the authenticated has been set to True (step 1005: Yes), the process proceeds to step 1003, whereas if the authenticated has been set to True, the process proceeds to step 1006. Through this process, the labels adjusted and authenticated by the user can be stored in the dataset (the high-level label dataset 207 and / or the low-level label dataset 205).

[0069] Next, the human annotation unit 104 determines whether or not the user has selected retraining the model (step 1006), and if it is determined that retraining the model has been selected (step 1006: Yes), the human annotation unit 104 instructs the hierarchical activity recognition model training unit 102 to execute the hierarchical activity recognition model training process 300 in order to retrain the model. On the other hand, if it is determined that retraining the model has not been selected (step 1006: No), the human annotation unit 104 advances the process to step 1001.

[0070] According to the above process, in the human annotation process 1000, a user can easily add or adjust a label to appropriately set a label for time-series data, in response to the label predicted by the machine annotation process 900. In addition, the model can be retrained using the set label, thereby improving the accuracy of label prediction by the model.

[0071] Next, the user interface will be described.

[0072] FIG. 11 is a diagram illustrating a user interface according to the first embodiment.

[0073] A display screen 1900 serving as a user interface includes a video display area 1901, a time-series data display area 1902, a high-level label display area 1903, a low-level label display area 1904, a video information display area 1911, and a label information display setting area 1920. On the display screen 1900, information is displayed based on the low-level labels L and high-level labels H output by the machine annotation unit 103, the human time-series data dataset 201, the low-level label dataset 205, and the high-level label dataset 207.

[0074] In the video display area 1901, video of video data of a worker working is displayed. This video data is an example of time series data. In the time series data display area 1902, time series data (sensor data) of sensor values ​​of a sensor attached to the worker working is displayed. The sensor data is an example of time series data. In the high level label display area 1903, each high level label (e.g., 1905) indicating a high level activity included in the time series data is displayed, for example, as a rectangle. In the low level label display area 1904, one or more low level labels (e.g., 1906) indicating a low level activity included in the high level activity of the high level label in the high level label display area 1903 are displayed, for example, as a rectangle. In the time series data display area 1902, the high level label display area 1903, and the low level label display area 1904, the horizontal axis indicates time.

[0075] The video information display area 1911 displays operation buttons and information (e.g., the time of the displayed video) regarding the video data of the video displayed in the video display area 1901. In this embodiment, the time series data display area 1902, the high level label display area 1903, and the low level label display area 1904 display a time indication bar 1907 indicating the portion corresponding to the video time.

[0076] The label information display setting area 1920 is an area for displaying and setting information about the low-level label selected in the low-level label display area 1904. In the example of Fig. 11, the label information display setting area 1920 displays information about the low-level label 1909 selected in the low-level label display area 1904.

[0077] The label information display setting area 1920 includes a start time display setting area 1915 , an end time display setting area 1914 , a label class display setting area 1913 , and an authenticated display setting area 1912 .

[0078] The start time display setting area 1915 displays the start time of the selected low-level label in a settable manner. In this embodiment, the time-series data display area 1902, the high-level label display area 1903, and the low-level label display area 1904 display a start time indication bar 1908 indicating the start time. The end time display setting area 1914 displays the end time of the selected low-level label in a settable manner. In this embodiment, the time-series data display area 1902, the high-level label display area 1903, and the low-level label display area 1904 display an end time indication bar 1910 indicating the end time. The label class display setting area 1913 displays the label class of the selected low-level label in a settable manner. The authenticated display setting area 1912 displays a value indicating whether the selected low-level label is authenticated or not in a settable manner.

[0079] In this embodiment, the user can make various settings for low-level labels by inputting (setting) information in each setting area of ​​the label information display setting area 1920. Also, in this embodiment, by selecting a high-level label in the high-level label display area 1903, a label information display setting area that displays information about the high-level label in a configurable manner is displayed. The label information display setting area for high-level labels has the same configuration as the label information display setting area 1920.

[0080] Next, the computer 10 constituting the labeling system 1 will be described.

[0081] FIG. 12 is a configuration diagram of a computer that constitutes the labeling system according to the first embodiment.

[0082] The computer 10 is an example of an information processing device, and is, for example, a PC (Personal Computer), a general-purpose server, a tablet computer, a mobile computer, etc. The computer 10 includes a communication interface (I / F) 11, one or more CPUs (Central Processing Units) 12, an input device 13, a storage device 14, a memory 15, and a display device 16.

[0083] The communication I / F 11 is, for example, an interface such as a wired LAN card or a wireless LAN card, and communicates with other devices via a network (not shown).

[0084] The CPU 12 is an example of a processor, and executes various processes according to programs stored in the memory 15 and / or the storage device 14.

[0085] The memory 15 is, for example, a RAM (RANDOM ACCESS MEMORY), and stores the programs executed by the CPU 12 and other necessary information.

[0086] The storage device 14 is, for example, a hard disk or a flash memory, and stores programs executed by the CPU 12, data used by the CPU 12, etc. In this embodiment, the storage device 14 stores programs and the like that realize the hierarchical activity recognition model training unit 102, the machine annotation unit 103, and the human annotation unit 104. The storage device 14 also stores a time series data dataset 201, a low level class probability dataset 203, a low level label dataset 205, a high level label dataset 207, etc.

[0087] The input device 13 is, for example, a mouse, a keyboard, etc., and accepts information input by a user. The display device 16 is, for example, a display, and displays and outputs a user interface including various types of information.

[0088] The labeling system 1 may be configured with the computer 10 and a computer used by a user (user computer: information processing device) connected to the computer 10 via a network. In this case, the processes executed by the computer 10 and the user computer may be arbitrarily separated, and for example, the processes executed by the input device 13 and the display device 16 may be executed by the user computer. Systems according to other embodiments described later may be configured with the computer 10 as described above, or the computer 10 and a user computer connected via a network.

[0089] Next, a labeling system according to a second embodiment will be described. Note that the same components as those in the labeling system according to the first embodiment will be denoted by the same reference numerals.

[0090] In the labeling system 1 according to the first embodiment, when a label is added or changed in the human annotation process 1000, the hierarchical activity recognition model training process 300 is executed again, and then the machine annotation process 900 is executed. However, in the labeling system 1A according to the second embodiment, the accuracy of labeling for time-series data can be improved without executing the hierarchical activity recognition model training process 300. This makes it possible to quickly improve the accuracy of labeling without executing the hierarchical activity recognition model training process 300, which takes a relatively long time.

[0091] FIG. 13 is a functional configuration diagram of a labeling system according to the second embodiment.

[0092] Compared to the labeling system 1, the labeling system 1A includes a human annotation unit 1102 instead of the human annotation unit 104, and further includes a machine annotation / partial reconstruction unit 1101.

[0093] The human annotation unit 1102 is a functional unit in which new processing has been added to the human annotation unit 104. Specific processing of the human annotation unit 1102 and the machine annotation / partial reconstruction unit 1101 will be described later.

[0094] Next, a machine annotation / part reconstruction process 1300 of the machine annotation / part reconstruction unit 1101 will be described.

[0095] Fig. 14 is a diagram for explaining the processing of the machine annotation / part reconstruction unit according to the second embodiment. Fig. 15 is a flowchart for explaining the machine annotation / part reconstruction processing according to the second embodiment.

[0096] The machine annotation and partial reconstruction unit 1101 inputs the low-level labels L of the low-level label dataset 1205, the high-level labels H of the high-level label dataset 1207, and the low-level class probability P of the low-level class probability dataset 203 (step 1301). Here, the low-level label dataset 1205 is a dataset in which changes made to the low-level label dataset 205 through processing by the human annotation unit 1102 are reflected, and the high-level label dataset 1207 is a dataset in which changes made to the high-level label dataset 207 through processing by the human annotation unit 1102 are reflected.

[0097] The machine annotation / part reconstruction unit 1101 deletes unauthenticated labels from the low-level label L (step 1302). Specifically, the machine annotation / part reconstruction unit 1101 sets the labels in the low-level label L for which the value of Authenticated 606 is True as the low-level label L. Here, the difference from the process of step 902 of the machine annotation unit 103 is that the labels for which the value of Authenticated 705 is False are not deleted from the high-level label H.

[0098] For each high-level label h in the high-level labels H, the machine annotation and partial reconstruction unit 1101 predicts a sequence of low-level labels N for that time interval by running the sequential model 206 corresponding to the high-level label h for the time interval corresponding to the high-level label h in the low-level class probability dataset 203. The machine annotation and partial reconstruction unit 1101 then sets the authenticated columns for these predicted low-level labels N to False (step 1303).

[0099] The machine annotation / part reconstruction unit 1101 mixes the low-level label N with the low-level label L to set it as a new low-level label L (step 1304). Then, the machine annotation / part reconstruction unit 1101 passes the low-level label L as an output to the human annotation unit 1102 to be reviewed in human annotation (step 1305).

[0100] Next, the overall processing including the human annotation process 1400 by the human annotation unit 1102 will be described.

[0101] Fig. 16 is a flowchart of the overall process according to the second embodiment. Note that the same reference numerals are used to denote the same parts as those in the overall process of Fig. 10.

[0102] In this embodiment, if it is determined in step 1006 that model retraining has not been selected (step 1006: No), the human annotation unit 1102 determines whether the high-level labels have been corrected (step 1401), and if the high-level labels have not been corrected (step 1401: No), the process proceeds to step 1001, whereas if the high-level labels have been corrected (step S1401: Yes), the process proceeds to step 1300.

[0103] In step 1300 , the machine annotation / part reconstruction unit 1101 executes a machine annotation / part reconstruction process 1300 .

[0104] According to the above process, when the high-level labels are modified, the machine annotation and partial reconstruction process 1300 can be performed without performing the hierarchical activity recognition model training process 300 for retraining the model. This makes it possible to improve the accuracy of labeling for time-series data in a short time without performing the hierarchical activity recognition model training process 300, which takes a relatively long time.

[0105] Next, a labeling system according to a third embodiment will be described. Note that the same components as those in the labeling system according to the first embodiment will be denoted by the same reference numerals.

[0106] Fig. 17 is a diagram for explaining the processing of the machine annotation unit according to the third embodiment. Fig. 18 is a flowchart of the machine annotation processing according to the third embodiment.

[0107] The labeling system according to the third embodiment is a system equipped with a machine annotation unit 1500 instead of the machine annotation unit 103 of the labeling system 1 according to the first embodiment. In addition to the functions of the machine annotation unit 103, the machine annotation unit 1500 includes an importance scoring unit 1501 and an importance scoring unit 1503. The importance scoring unit 1501 scores the importance of the high-level label output from the high-level model 204, and outputs the importance data to a high-level label priority dataset 1502. The importance scoring unit 1503 scores the importance of the low-level label output from the sequential model 206, and outputs the importance data to a low-level label priority dataset 1504.

[0108] In the machine annotation process, the machine annotation unit 1500 inputs the low-level label L of the low-level label dataset 205, the high-level label H of the high-level label dataset 207, and the low-level class probability P of the low-level class probability dataset 203 (step 1601).

[0109] The machine annotation unit 1500 deletes unauthenticated labels from the high-level label H and the low-level label L (step 1602). Specifically, the machine annotation unit 1500 sets the labels in the high-level label H for which the value of Authenticated 705 is True as the high-level label H, and sets the labels in the low-level label L for which the value of Authenticated 606 is True as the low-level label L.

[0110] The machine annotation unit 1500 inputs the low-level class probability P obtained from the time series data into the high-level model 204 to predict high-level labels R for unlabeled time intervals in the time series data, and sets Authenticated to False for all predicted high-level labels R (step 1603).

[0111] The importance scorer 1501 of the machine annotation unit 1500 calculates the importance of each label of the high-level labels R as I. For example, the importance scorer 1501 may determine the importance indicating the certainty of the model by calculating the entropy of the class probability for each class output from the high-level model (e.g., a classifier in the model) for the target label. A high entropy is calculated for a target label indicating high uncertainty. The importance scorer 1501 assigns an importance of 0 to authenticated labels present in the high-level label dataset 207, and sets the importance as Q (step 1604).

[0112] The machine annotation unit 1500 mixes the high-level label R with the high-level label H to set a new high-level label H, and mixes the importance Q with the importance I to set a new importance I (step 1605). The importance I is stored in the high-level label priority dataset 1502.

[0113] For each high-level label h in the high-level labels H, the machine annotation unit 1500 predicts the sequence of low-level labels N for that time interval by running the sequential model 206 corresponding to high-level label h on an unlabeled time interval in the time series data. The machine annotation unit 1500 then sets the Authenticated column to False for all of these predicted low-level labels N (step 1606).

[0114] The importance scorer 1503 of the machine annotation unit 1500 calculates the importance of each label of the low-level labels N as J. For example, the importance scorer 1503 may determine the importance indicating the certainty of the model by calculating the entropy of the class probability for each class output from the sequential model 206 (e.g., a classifier in the model) for the target label. A high entropy is calculated for a target label indicating high uncertainty. The importance scorer 1503 assigns an importance of 0 to an authenticated label present in the low-level label dataset 205, giving it an importance of G (step 1607).

[0115] The machine annotation unit 1500 mixes the low-level label N with the low-level label L to set a new low-level label L, and mixes the importance G with the importance J to set a new importance J (step 1608). The importance J is stored in the low-level label priority dataset 1504.

[0116] The machine annotation unit 1500 then passes the low-level label L, the high-level label H, the importance I, and the importance J as output to the human annotation unit 104 for review in human annotation (step 1609).

[0117] After that, the human annotation unit 104 displays each label together with the importance of the label through a user interface, allowing the user to specify the importance of the displayed labels and easily and appropriately grasp the labels that require attention.

[0118] Next, the configuration of the high-level label priority data set 1502 will be described.

[0119] FIG. 19 is a diagram showing the configuration of a high-level label priority data set according to the third embodiment.

[0120] The high-level label priority dataset 1502 stores an entry for each high-level label in the high-level label dataset 207. An entry in the high-level label priority dataset 1502 includes columns for a high-level label 1701 and an importance 1702.

[0121] An index indicating a high-level label corresponding to an entry is stored in high-level label 1701. Importance 1702 stores the importance calculated by importance rating unit 1501 for the high-level label corresponding to the entry.

[0122] Next, the configuration of the low-level label priority data set 1504 will be described.

[0123] FIG. 20 is a diagram showing the configuration of a low-level label priority data set according to the third embodiment.

[0124] The low-level label priority dataset 1504 stores an entry for each low-level label in the low-level label dataset 205. An entry in the low-level label priority dataset 1504 includes columns for a low-level label 1801 and importance 1802.

[0125] An index indicating a low-level label corresponding to an entry is stored in low-level label 1801. Importance 1802 stores the importance calculated by importance rating unit 1503 for the low-level label corresponding to the entry.

[0126] Next, an activity recognition system 2 according to a fourth embodiment will be described. Note that the same components as those in the labeling system according to the first embodiment will be denoted by the same reference numerals.

[0127] The activity recognition system 2 according to the fourth embodiment detects errors in activities contained in input time series data using machine learning models trained by the labeling system 1 according to the first embodiment (the low-level model 202, the high-level model 204, and the sequential model 206 after training).

[0128] Fig. 21 is a functional configuration diagram of an activity recognition system according to the fourth embodiment Fig. 22 is a flowchart of an error detection process according to the fourth embodiment.

[0129] The activity recognition system 2 receives the time series data S of the time series data set 2001 (step 2101). Here, the time series data set 2001 is a data set of unlabeled time series data to be used for activity recognition, and includes video data of a person performing an activity or time series sensor data obtained by a sensor attached to the person performing an activity. Each piece of time series data may be data obtained in real time. The configuration of the time series data set 2001 itself is the same as that of the time series data set 201.

[0130] The activity recognition system 2 inputs the time series data S to the low-level model 202 to generate a low-level class probability P at each time for the time series data S (step 2102). The generated low-level class probability P is stored in a low-level class probability dataset 2002. The configuration of the low-level class probability dataset 2002 itself is similar to that of the low-level class probability dataset 203.

[0131] The activity recognition system 2 predicts (generates) a high-level label H for the time-series data by inputting the low-level class probability P obtained from the time-series data to the high-level model 204, and stores the generated high-level label H in the high-level label dataset 2003 (step 2103). The configuration of the high-level label dataset 2003 itself is the same as that of the high-level label dataset 207.

[0132] For each high-level label h of the high-level labels H, the activity recognition system 2 predicts a sequence of low-level labels L for the time interval corresponding to the high-level label h in the low-level class probability dataset 2002 by executing the sequential model 206 corresponding to the high-level label h for that time interval, and stores the generated low-level labels L in the low-level label dataset 2004 (step 2104). The configuration of the low-level label dataset 2004 itself is similar to that of the low-level label dataset 205.

[0133] The error detector 2005 of the activity recognition system 2 calculates the probability of an error or anomaly occurring at each time in the time series data based on the output from the machine learning model (step 2105). The error detector 2005 may analyze a matrix of emission probabilities output from a forward-backward algorithm executed as part of the HMM used as the sequential model 206. In this case, the matrix of emission probabilities gives the probability in the HMM of an observed vector being emitted in the low-level class probability dataset 203 used as input for the sequential model. The error detector 2005 detects unexpected observations by calculating the entropy of the emission probabilities at each time interval, with high entropy indicating a potential error or anomaly in the time series data, thereby ensuring a high probability of error detection at that time. The error detector 2005 assigns 0 as the probability of error detection to all times that have not been run through the sequential model, since they do not have a corresponding high-level label.

[0134] Next, the activity recognition system 2 outputs the error detection probability Z (step 2106). After that, the activity recognition system 2 may display information based on the error detection probability through a user interface. Also, the activity recognition system 2 may determine whether or not an abnormal activity is included based on the error detection probability, and display the result through a user interface.

[0135] Next, the configuration of the error detection probability data set 2006 will be described.

[0136] FIG. 23 is a configuration diagram of an error detection probability data set according to the fourth embodiment.

[0137] The error detection probability data set 2006 stores an entry for each time. The entry of the error detection probability data set 2006 includes columns of a timestamp 2201 and an error detection probability 2202.

[0138] The timestamp 2201 stores a timestamp indicating the time corresponding to the entry. For example, if there are H samples in total in the time series data set 201, the timestamp of the first sample is denoted by T1, the timestamp of the last sample is denoted by T2, H In this embodiment, the error detection probability is generated one-to-one with respect to the original sample of the time series data, but this is not limited to this. The error detection probability 2202 stores the error detection probability in the time series data calculated by the error detection unit 2005 at the time indicated by the time stamp corresponding to the entry.

[0139] Next, a teaching material generating system 3 according to the fifth embodiment will be described. Note that the same components as those in the labeling system according to the first embodiment will be denoted by the same reference numerals.

[0140] The teaching material generation system 3 according to the fifth embodiment generates teaching materials from time-series data using machine learning models trained by the labeling system 1 according to the first embodiment (the low-level model 202, the high-level model 204, and the sequential model 206 after training).

[0141] Fig. 24 is a functional configuration diagram of a teaching material generating system according to the fifth embodiment Fig. 25 is a flowchart of a teaching material generating process according to the fifth embodiment.

[0142] Steps 2401 to 2404 performed by the teaching material generation system 3 are similar to steps 2101 to 2104 performed by the activity recognition system 2 in the fourth embodiment.

[0143] The teaching material generation unit 2301 of the teaching material generation system 3 uses a machine learning model to generate a label class {HC1, HC2, . . . , HC K}, the expected low-level step sequences (i.e., low-level label dataset 205) for label class c) from {c,d,e,f,e,f,g ...

[0144] According to this teaching material generating system 3, it is possible to allow the user to properly grasp a high-level label class and a low-level label class that belongs to the high-level label class.

[0145] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention.

[0146] For example, in the above embodiment, the machine learning model that takes time series data as input and outputs high-level labels and low-level labels includes the low-level model 202, the high-level model 204, and the sequential model 206. However, the present invention is not limited to this, and may be composed of one model or multiple other models.

[0147] In addition, the activity recognition system 2 according to the fourth embodiment or the teaching material generation system 3 according to the fifth embodiment may be provided with a configuration for training the machine learning model in the first to third embodiments (such as the hierarchical activity recognition model training unit 102) to train the machine learning model.

[0148] In the above embodiment, a part or all of the processing performed by the CPU may be performed by a hardware circuit. The program in the above embodiment may be installed from a program source. The program source may be a program distribution server or a recording medium (e.g., a portable recording medium). [Explanation of symbols]

[0149] 1,1A... labeling system, 2... activity recognition system, 3... teaching material generation system, 10... computer, 12... CPU, 102... hierarchical activity recognition model training unit, 103... machine annotation unit, 104... human annotation unit, 202... low-level model, 204... high-level model, 206... sequential model

Claims

1. A labeling system for labeling activities included in time-series data, comprising: The activities include low-level activities and high-level activities that are composed of a plurality of the low-level activities; The labeling system comprises: A processor; A machine learning model that receives time-series data as input and outputs high-level labels that are labels indicating high-level activities included in the time-series data and low-level labels that indicate a plurality of low-level activities included in the high-level activities, The processor, Using the machine learning model, outputting high-level and low-level labels for high-level and low-level activities included in the target time series data; Displaying high-level and low-level labels of high-level and low-level activities contained in the output time series data of the target; Accepting an input of high-level labels included in the time series data; training the machine learning model such that the time series data includes high-level activity corresponding to the accepted high-level labels; Using the trained machine learning model, high-level and low-level labels for high-level and low-level activities included in the target time series data are output. Labeling system.

2. The machine learning model is a low-level model that receives time-series data as input and outputs a low-level activity probability in a time series, the low-level activity probability being a probability that a plurality of low-level activities in the time-series data belong to a presumed low-level activity type; a high-level model that receives low-level activity probabilities in a time series and outputs high-level labels that indicate high-level activities included in the time-series data of the low-level activity probabilities; a sequential model that receives low-level activity probabilities and high-level labels as input and outputs low-level labels indicating a plurality of low-level activities included in the time series data; The processor, training the high-level model assuming that the time series data contains high-level activity corresponding to the accepted high-level labels; Using the trained high-level model, high-level labels and low-level labels for the high-level and low-level activities contained in the target time series data are output. The labeling system of claim 1 .

3. The processor, When an input for the high-level label is received from a user, the received high-level label is used as an input for the sequential model without training the high-level model, and high-level labels and low-level labels for high-level activities and low-level activities included in the target time-series data are output. The labeling system of claim 2 .

4. The high-level model outputs a plurality of high-level labels and a significance of the presence of the high-level activity corresponding to the high-level label in the target time series data; The processor, The high-level label and the importance are output to the user in a recognizable manner. The labeling system of claim 2 .

5. The processor, A user input regarding the high-level labels included in the time-series data is received, such as a specification of a new high-level label for a high-level activity included in the time-series data, or an instruction to change the output high-level label. The labeling system of claim 2 .

6. The processor, receiving, from the user, a designation of a new low-level label for the low-level label activity included in the time-series data, or an instruction to change the output low-level label; training the low-level model assuming that the time series data includes low-level activity corresponding to the accepted low-level labels; The labeling system of claim 5 .

7. The labeling system comprises: Consists of a single information processing device The labeling system of claim 1 .

8. The labeling system comprises: The system is configured by a server and an information processing device connected to the server via a network. The labeling system of claim 1 .

9. The time series data is video data capturing human activity or sensor data from a sensor that can be worn by a human. The labeling system of claim 1 .

10. An activity recognition system for recognizing activities included in time series data, comprising: The activities include low-level activities and high-level activities that are composed of a plurality of the low-level activities; The activity recognition system includes: A processor; a machine learning model that is trained by the labeling system according to claim 1 and that receives time-series data as input and outputs high-level labels that are labels indicating high-level activities included in the time-series data and low-level labels that indicate a plurality of low-level activities included in the high-level activities; The processor, This is video data capturing human activity or sensor data from sensors that can be worn by humans. Accepts time series data input sequentially, Using the machine learning model, sequentially outputting high-level labels and low-level labels for high-level activities and low-level activities included in the target time-series data; Based on the sequentially output high-level and low-level labels, human activities are sequentially recognized. Activity recognition system.

11. The processor, Determining whether the recognized human activity is an abnormal activity. The activity recognition system of claim 10.

12. A teaching material generation system for generating teaching material showing a relationship between a high-level activity and a low-level activity in an activity including a low-level activity and a high-level activity composed of a plurality of the low-level activities, comprising: The teaching material generation system includes: A processor; a machine learning model that is trained by the labeling system according to claim 1 and that receives time-series data as input and outputs high-level labels that are labels indicating high-level activities included in the time-series data and low-level labels that indicate a plurality of low-level activities included in the high-level activities; The processor, Using the machine learning model, outputting high-level and low-level labels for high-level and low-level activities included in a given time series data; Based on the high-level and low-level labels of the high-level and low-level activities contained in the output time series data, a teaching material is generated that summarizes information showing the correspondence between the high-level and low-level labels. Teaching material generation system.

Images