Learning device, learning method, and learning program

Abstract

In order to enable execution of machine learning for obtaining a high-accuracy model while reducing labeling cost, a learning device (1) comprises: a class association unit (11) that associates, with a time-series data set for training, a class indicated by a label assigned to data included in the time-series data set for training; a class propagation unit (12) that associates, with the time-series data set for training, a class associated with another time-series data set for training on the basis of the similarity between time-series data sets for training; a pseudo label assigning unit (13) that assigns a pseudo label to data to which the labels included in the time-series data sets for training are not assigned; and a learning unit (14) that executes machine learning by using the time-series data set for training including the data to which the pseudo label is assigned. The pseudo label assigning unit (13) limits pseudo labels to be assigned to data included in a time-series data set for training on the basis of the class associated with the time-series data set for training.

Description

Learning device, learning method, and learning program

[0001] The present invention relates to a learning device, a learning method, and a learning program.

[0002] In machine learning, it is necessary to reduce the cost of collecting training data. For example, when learning time-series data segment recognition tasks, reducing the high cost of labeling is a challenge.

[0003] The time-series data interval recognition task is a task to classify data x at each time instant into a class yi yi∈Y={yi, y2, ..., yK} when given time-series data X X=[x1, x2, ..., xT], where yi may be one class or multiple classes.

[0004] To train a time-series data interval recognition task, a correct class label yi is generally given as training data for all time-series data xi. The class label in this case is called a full label. The label assignment cost of a full label is very high.

[0005] Therefore, in order to reduce the labeling cost in the time-series data interval recognition task, labeling only some of the labels has been considered.

[0006] That is, among the time series data X X = [x1, x2, ..., xT], class yi yi∈Y = {y1, y2, ..., yK} is given as training data only for elements xi of a subset X^ ⊂ X of X. Such class labels are called weak labels. Generally, the labeling cost of weak labels is lower than that of full labels.

[0007] For example, Non-Patent Document 1 describes a technique for learning a video action segment recognition task, which is an example of a time-series data segment recognition task, using weak labels. In this technique, a model is first trained using timestamp-type labels. Next, using the inference results of the trained model, pseudo-labels are assigned to areas near data at labeled times, and these are used for learning together. Pseudo-labels are labels that are assigned pseudo-wise to data at times that have not been labeled.

[0008] Ma et al., SF-Net: Single-Frame Supervision for Temporal Action Localization, ECCV 2020

[0009] However, in the conventional technology of Non-Patent Document 1, pseudo labels are assigned only to data in the vicinity of the time at which the label is assigned in time series data. Therefore, pseudo labels cannot be assigned to time regions distant from the time at which the label is assigned or to time series data to which no label has been assigned in the first place, and there is a limit to the improvement of inference accuracy.

[0010] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of its purpose is to enable learning of a highly accurate machine learning model that infers into which class data at each time point in time series data is classified while reducing labeling costs.

[0011] a pseudo-label assignment unit that assigns, to each piece of teacher time series data, a pseudo-label indicating a class into which the machine learning model has classified data that is not assigned a label, based on the class associated with each piece of teacher time series data; a pseudo-label assignment unit that assigns, to each piece of teacher time series data, a pseudo-label indicating a class into which the machine learning model has classified data that is not assigned a label, based on the class associated with each piece of teacher time series data; and a learning unit that trains the machine learning model using the plurality of teacher time series data including data that has been assigned the pseudo-label; and the pseudo-label assignment unit that restricts the pseudo-label to be assigned to data included in the teacher time series data, based on the class associated with each piece of teacher time series data.

[0012] A learning method according to one aspect of the present invention is a learning method for machine learning, using a plurality of teacher time series data, to create a machine learning model that infers into which class data at each time point in time series data is classified, wherein some data included in the plurality of teacher time series data are assigned labels indicating the classes, and the learning method includes the following steps: a class association process for associating each teacher time series data with the class indicated by the label assigned to the data included in the teacher time series data; a class propagation process for associating at least one teacher time series data with at least some of the classes associated with other teacher time series data based on the similarity between the teacher time series data; a pseudo-label assignment process for assigning, for each teacher time series data, pseudo-labels indicating the class into which the machine learning model has classified data that is not assigned a label included in the teacher time series data; and a learning process for machine learning the machine learning model using the plurality of teacher time series data including data assigned the pseudo-labels, wherein the pseudo-label assignment process restricts the pseudo-labels to be assigned to data included in the teacher time series data based on the class associated with each teacher time series data.

[0013] A learning program according to one aspect of the present invention is a learning program that causes a computer to perform machine learning on a machine learning model that infers into which class data at each time point in time series data is classified, using a plurality of teacher time series data, wherein some data included in the plurality of teacher time series data are assigned labels indicating the classes, and the program includes a class matching process that matches each teacher time series data with the class indicated by the label assigned to the data included in the teacher time series data, and a class matching process that matches at least one teacher time series data with other teacher time series data based on the similarity between the teacher time series data. a pseudo-labeling process for assigning pseudo-labels indicating classes into which the machine learning model has classified data to data that is not assigned a label included in the teacher time series data; and a learning process for training the machine learning model by using the plurality of teacher time series data including data to which the pseudo-labels have been assigned, wherein the pseudo-labeling process limits the pseudo-labels to be assigned to data included in the teacher time series data based on the classes associated with each teacher time series data.

[0014] This enables learning of a highly accurate machine learning model that infers which class data at each time point in time series data should be classified into while reducing labeling costs.

[0015] FIG. 1 is a block diagram showing the configuration of a learning device according to exemplary embodiment 1 of the present invention. FIG. 2 is a flow diagram showing the flow of a learning method according to exemplary embodiment 1 of the present invention. FIG. 3 is a block diagram showing the configuration of a learning device according to exemplary embodiment 2 of the present invention. FIG. 4 is a flow diagram showing the flow of a learning method according to exemplary embodiment 2 of the present invention. FIG. 5 is a block diagram showing the configuration of a learning device according to exemplary embodiment 3 of the present invention. FIG. 6 is a flow diagram showing the flow of a learning method according to exemplary embodiment 3 of the present invention. FIG. 7 is a block diagram showing an example of the hardware configuration of each device according to each exemplary embodiment of the present invention.

[0016] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0017] (Configuration of Learning Device 1) The learning device 1 according to this exemplary embodiment uses multiple teacher time-series data to perform machine learning on a machine learning model that infers which class data at each time point in the time-series data falls into. Some data included in the multiple teacher time-series data are assigned labels indicating classes. The class labels may be, for example, a single label assigned to each piece of data at each time point in the time-series data, or multiple labels may be assigned. The multiple teacher time-series data may include, for example, multiple independent pieces of data, or multiple pieces of time-series data that are related to each other and are generated by dividing a single piece of time-series data into multiple pieces. The time-series data may be fully labeled, partially labeled, or completely unlabeled. The time-series data may be, for example, video or audio. An example of a time-series data segment recognition task that is the subject of machine learning is the task of inferring classes representing each activity in a video and the segments involved in detecting activity segments in a video. Another example of a time-series data segment recognition task is a task for detecting event segments in audio, in which audio is classified at each time point in time-series audio data. The data at each time point in the time-series data corresponds to, for example, frames in video or audio.

[0018] The configuration of a learning device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the learning device 1. As shown in Fig. 1, the learning device 1 includes a class mapping unit 11, a class propagation unit 12, a pseudo-label assignment unit 13, and a learning unit 14.

[0019] The class associating unit 11 associates each piece of teacher time-series data with a class indicated by a label assigned to data included in the teacher time-series data. Class associating refers to associating a class with the entirety of each piece of time-series data. The class associating unit 11 assigns the time-series data with a class indicated by a label assigned to the data included in the time-series data. For example, if the time-series data is a video, this refers to assigning each video with a class indicated by a label assigned to a frame within the video. In one aspect, if a class corresponding to time-series data is directly specified, the class associating unit 11 may assign the class to the time-series data.

[0020] The class propagation unit 12 associates at least some of the classes associated with other teacher time series data with at least one teacher time series data based on the similarity between the teacher time series data. The similarity indicates how similar the features of the time series data are to each other.

[0021] In one aspect, the features of each time-series data unit are represented by feature quantities. When the time-series data is a video, the feature quantities for the video are, for example, the average of the feature quantities for all frames. In the space representing the feature quantities, the closer the positions of the feature quantities are, the higher the similarity is determined to be. The class propagation unit 12 assumes that time-series data with sufficiently high similarity to each other have similar classes, and associates all or part of the classes associated with one time-series data with the other time-series data.

[0022] For example, the class propagation unit 12 may select K classes (where K is a natural number and is less than or equal to the total number of time series data) in descending order of similarity from time series data whose classes within the time series data are known, and assign all or some of the classes within the time series data to the selected classes.

[0023] Furthermore, for example, when the class propagation unit 12 focuses on certain time series data whose class within the time series data is unknown, the label of the class may be considered to be reliable and a valid class only if the same class label is assigned from multiple labeled time series data.

[0024] Furthermore, for example, when the class propagation unit 12 focuses on certain time series data whose class within the time series data is unknown, and there are multiple time series data whose class within the time series data is known and has similarity, the known class in the time series data with the largest total number may be assigned to the time series data whose class within the time series data is unknown.

[0025] Furthermore, for example, the class propagation unit 12 may assign a class within the time-series data to time-series data whose similarity is sufficiently close, weighted by the similarity.

[0026] Furthermore, for example, the class propagation unit 12 may further assign the class within the propagated time series data to other time series data.

[0027] The pseudo-labeling unit 13 assigns pseudo-labels indicating classes into which the machine learning model has classified unlabeled data included in each of the training time-series data. In the time-series data, pseudo-labels based on data that has already been assigned labels can be assigned to both unlabeled data and labeled data.

[0028] The pseudo label assigning unit 13 limits the pseudo labels to be assigned to data included in each teacher time-series data based on the class associated with the teacher time-series data. The pseudo label assigning unit 13 limits the pseudo labels to be assigned based on the class already associated with the time-series data. Examples of the conditions for the limitation include the constraint conditions in exemplary embodiment 2 described below.

[0029] By restricting the pseudo-labels assigned to the data contained in each time series data based on the classes already associated with that time series data or time series data similar to that time series data, it is possible to prevent the assignment of pseudo-labels of incorrect classes, such as classes that do not exist in the time series data.

[0030] The learning unit 14 trains a machine learning model using a plurality of training time-series data sets including data sets to which pseudo-labels have been assigned.

[0031] (Flow of Learning Method S1) The learning device 1 configured as above executes the learning method S1 according to this exemplary embodiment.

[0032] The learning method S1 uses a plurality of training time-series data sets to train a machine learning model that infers into which class data at each time point in the time-series data sets falls, where some data sets included in the training time-series data sets are assigned labels indicating the classes.

[0033] The flow of the learning method S1 will be described with reference to FIG. 2 . FIG. 2 is a flow chart showing the flow of the learning method S1. As shown in FIG. 2 , the learning method S1 includes a class matching step S11, a class propagation step S12, a pseudo-label assignment step S13, and a learning step S14. In the class matching step S11, the class matching unit 11 matches each piece of teacher time series data with a class indicated by a label assigned to data included in the teacher time series data. In the class propagation step S12, the class propagation unit 12 matches at least one piece of teacher time series data with at least a portion of the classes associated with other teacher time series data based on the similarity between the teacher time series data. In the pseudo-label assignment step S13, the pseudo-label assignment unit 13 assigns, for each piece of teacher time series data, a pseudo label indicating the class into which the machine learning model has classified the data, to data that is not assigned a label included in the teacher time series data. In the pseudo-labeling step S13, the pseudo-labels to be assigned to the data included in each of the training time-series data are limited based on the classes associated with the training time-series data. In the learning step S14, the learning unit 14 trains a machine-learning model using a plurality of training time-series data including data to which pseudo labels have been assigned.

[0034] As described above, the learning device 1 and learning method S1 according to this exemplary embodiment can prevent the assignment of pseudo labels of incorrect classes, such as classes that do not exist in the teacher time-series data. As a result, the number and variety of assigned pseudo labels increases, which is expected to result in high inference accuracy.

[0035]

[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0036] (Configuration of Learning Device 10) The configuration of a learning device 10 according to a second exemplary embodiment of the present invention will be described with reference to FIG. 3. FIG. 3 is a block diagram showing the functional configuration of the learning device 10. As shown in FIG. 3, the learning device 10 includes a control unit 110 and a storage unit 120. The control unit 110 controls each unit of the learning device 10. The control unit 110 includes a class association unit 11, a class propagation unit 12, a pseudo-label assignment unit 13, a learning unit 14, an inference unit 15, a feature calculation unit 16, a similarity calculation unit 17, and a constraint condition assignment unit 18. The storage unit 120 stores various data used by the control unit 110. For example, the storage unit 120 stores teacher time-series data TD and a machine learning model MM.

[0037] The class associating unit 11 associates each piece of teacher time-series data TD with a class indicated by a label assigned to data included in the teacher time-series data TD. Class associating refers to associating a class with each piece of time-series data as a whole. The class associating unit 11 assigns, to the time-series data, a class indicated by a label assigned to the data included in the time-series data. For example, if the time-series data is a video, this refers to assigning, to each video, a class indicated by a label assigned to a frame within the video. In one aspect, if a class corresponding to time-series data is directly specified, the class associating unit 11 may assign the class to the time-series data.

[0038] The class propagation unit 12 associates at least some of the classes associated with other teacher time series data TD with at least one teacher time series data TD based on the similarity between the teacher time series data TD. The similarity indicates how similar the features of the time series data are to each other.

[0039] In one aspect, the features of each time-series data unit are represented by feature quantities. When the time-series data is a video, the feature quantities for the video are, for example, the average of the feature quantities for all frames. In the space representing the feature quantities, the closer the positions of the feature quantities are, the higher the similarity is determined to be. The class propagation unit 12 assumes that time-series data with sufficiently high similarity to each other have similar classes, and associates all or part of the classes associated with one time-series data with the other time-series data.

[0040] For example, the class propagation unit 12 may associate all classes associated with the first time series data with second time series data whose similarity to the first time series data is equal to or greater than a predetermined threshold. Alternatively, the class propagation unit 12 may associate some of the classes associated with the first time series data with second time series data whose similarity to the first time series data is equal to or greater than a predetermined threshold. For example, a feature may be generated for each class of time series data, and the class may be propagated between time series data whose similarity to the feature for the class is equal to or greater than a predetermined threshold. That is, given time series data having classes A, B, and C, if the similarity of the feature for class A is sufficiently high, the class A may be assigned, and if the similarity of the feature for class C is sufficiently low, the class C may not be assigned. For example, the feature for each class may be calculated using a machine learning model that receives input of each data item in the time series data and outputs a feature, and that is trained so that the output feature is large when data to which each class is assigned is input.

[0041] Furthermore, for example, the class propagation unit 12 may select K classes (where K is a natural number and is equal to or less than the total number of time series data) in descending order of similarity from time series data whose classes within the time series data are known, and assign all or some of the classes within the time series data to the selected classes.

[0042] Furthermore, for example, when the class propagation unit 12 focuses on certain time series data whose class within the time series data is unknown, the label of the class may be considered to be reliable and a valid class only if the same class label is assigned from multiple labeled time series data.

[0043] Furthermore, for example, when the class propagation unit 12 focuses on certain time series data whose class within the time series data is unknown, and there are multiple time series data whose class within the time series data is known and has similarity, the known class in the time series data with the largest total number may be assigned to the time series data whose class within the time series data is unknown.

[0044] Furthermore, for example, the class propagation unit 12 may assign a class within the time-series data to time-series data whose similarity is sufficiently close, weighted by the similarity.

[0045] Furthermore, for example, the class propagation unit 12 may further assign the class within the propagated time series data to other time series data.

[0046] The pseudo-labeling unit 13 assigns pseudo-labels indicating classes into which the machine learning model MM has classified unlabeled data included in each of the teacher time-series data TD. In the time-series data, pseudo-labels based on data that has already been assigned labels can be assigned to both unlabeled data and labeled data.

[0047] The pseudo label assigning unit 13 limits the pseudo labels to be assigned to data included in each teacher time-series data TD based on the class associated with the teacher time-series data TD. The pseudo label assigning unit 13 limits the pseudo labels to be assigned based on the class already associated with the time-series data. Examples of the conditions for the limitation include the constraint conditions in exemplary embodiment 2 described below.

[0048] By restricting the pseudo-labels assigned to the data contained in each time series data based on the classes already associated with that time series data or time series data similar to that time series data, it is possible to prevent the assignment of pseudo-labels of incorrect classes, such as classes that do not exist in the time series data.

[0049] The learning unit 14 performs machine learning on the machine-learning model MM using a plurality of teacher time-series data TD including data to which pseudo labels are assigned. The learning unit 14 may further include a configuration for calculating a loss using, as input, labels originally assigned to the teacher time-series data TD, pseudo labels assigned to the teacher time-series data TD, and inference results, and updating parameters of the machine-learning model MM using the loss as input. The loss refers to the magnitude of deviation between the labels originally assigned to the teacher time-series data TD or the pseudo labels assigned to the teacher time-series data TD and the inference results.

[0050] The inference unit 15 infers into which class the data at each time of the teacher time series data TD is classified.

[0051] The feature calculation unit 16 calculates feature values ​​for each piece of training time-series data TD on a time-series data basis. For example, the feature values ​​may be the output results of a pre-trained model, color features, or meta information. The meta information may be, for example, the acquisition time of the time-series data or the acquisition location of the time-series data. Furthermore, for example, if the time-series data is a video, the angle of view of the video acquisition camera may be used as the feature value.

[0052] Furthermore, for example, when time-series data is passed through a neural network, the feature calculation unit 16 may calculate feature amounts from values ​​representing the features of each piece of data at each time instant in the time-series data, which are output from the intermediate and final layers of the neural network. Alternatively, the feature amounts may be calculated after performing a pooling process such as averaging on the output values. Furthermore, during pooling, the pooling may be weighted by a prediction score or the like. Furthermore, the output values ​​may be passed through yet another neural network, and, for example, metric learning or contrastive learning may be performed in that space.

[0053] Furthermore, for example, the feature calculation unit 16 may calculate the feature from the time ratio of the time-series data section estimated from the inference result (for example, if the time-series data section is an action section in a video, what is the time ratio of each action). In this case, for example, the feature may be calculated so that it can be determined that the similarity between videos is high if the action time ratios in the videos are similar.

[0054] The similarity calculation unit 17 calculates the similarity between the training time-series data TD using the feature amount. For example, the similarity calculation may use cosine similarity, Euclidean distance, Manhattan distance (L1 norm), or Kullback-Leibler divergence (KL divergence).

[0055] The constraint condition assigning unit 18 assigns constraint conditions that limit the class of the pseudo label to the class of the label that originally exists in the teacher time series data TD or the class of the label in the teacher time series data TD that has been obtained by being assigned by the class propagation unit.

[0056] For example, the constraint condition assigning unit 18 may set a constraint condition that restricts the assignment of pseudo labels only to classes in the training time-series data TD, and does not assign pseudo labels to other classes.

[0057] Furthermore, for example, when the pseudo label assignment unit 13 assigns a pseudo label to data whose inference score exceeds a pseudo label threshold among the teacher time-series data TD that satisfies the constraint conditions, the constraint condition assignment unit 18 may assign different pseudo label thresholds to classes that are the same as the class of the label in the teacher time-series data TD and other classes.

[0058] Furthermore, for example, the constraint condition assigning unit 18 may change the constraint conditions on the teacher time-series data TD according to the progress of machine learning. Examples of the changes include removing or relaxing the constraint conditions.

[0059] (Flow of Learning Method S10) The learning device 10 configured as described above executes the learning method S10 according to this exemplary embodiment. The flow of learning method S10 will be described with reference to Fig. 4. Fig. 4 is a flow chart showing the flow of learning method S10. As shown in Fig. 4, learning method S10 includes steps S101 to S108.

[0060] In the class association step S101, the class association unit 11 associates each piece of teacher time series data TD with a class indicated by a label assigned to data included in the teacher time series data TD.

[0061] In the inference step S102, the inference unit 15 infers into which class the data at each time point of the teacher time series data TD is classified.

[0062] In the feature calculation step S103, the feature calculation unit 16 calculates the feature of each piece of teacher time series data TD for each time series data unit.

[0063] In the similarity calculation step S104, the similarity calculation unit 17 calculates the similarity between the teacher time-series data TD using the feature amount.

[0064] In the class propagation step S105, the class propagation unit 12 associates at least one of the teacher time series data TD with at least some of the classes associated with other teacher time series data TD based on the similarity between the teacher time series data TD.

[0065] In the constraint condition assignment step S106, the constraint condition assignment unit 18 assigns constraint conditions that limit the class of the pseudo label to the class of the label that originally exists in the teacher time series data TD or the class of the label in the teacher time series data TD that has been assigned by the class propagation unit.

[0066] In the pseudo-label assignment step S107, the pseudo-label assignment unit 13 assigns, for each piece of teacher time-series data TD, a pseudo-label indicating the class into which the machine learning model MM has classified the unlabeled data included in the teacher time-series data TD. Note that the pseudo-label assignment step S107 limits the pseudo-labels to be assigned to the data included in the teacher time-series data TD based on the class associated with each piece of teacher time-series data TD.

[0067] In a learning step S108, the learning unit 14 performs machine learning to generate a machine-learning model MM using a plurality of pieces of teacher time-series data TD including data to which pseudo labels have been assigned. The learning unit S108 may further include a configuration for calculating a loss using, as input, for example, labels originally assigned to the teacher time-series data TD, pseudo labels assigned to the teacher time-series data TD, and the result of inference, and for updating parameters of the machine-learning model MM using the loss as input.

[0068] As described above, the learning device 10 and learning method S10 according to this exemplary embodiment impose constraints on the pseudo labels, thereby making it possible to prevent the assignment of pseudo labels of incorrect classes, such as classes that do not exist in the teacher time-series data TD. As a result, the number and variety of assigned pseudo labels increases, which is expected to result in higher inference accuracy.

[0069]

[0033] A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0070] (Configuration of Learning Device 20) The configuration of a learning device 20 according to a third exemplary embodiment of the present invention will be described with reference to FIG. 5. FIG. 5 is a block diagram showing the functional configuration of the learning device 20. As shown in FIG. 5, the learning device 20 includes a control unit 210 and a storage unit 220. The control unit 210 controls each unit of the learning device 20. The control unit 210 includes a feature acquisition unit 21, a clustering unit 22, a data selection unit 23, a label acquisition unit 24, a class association unit 25, a class propagation unit 26, a pseudo-label assignment unit 27, and a learning unit 28. The storage unit 220 stores various data used by the control unit 210. For example, the storage unit 220 stores teacher time-series data TD and a machine learning model MM.

[0071] The feature amount acquiring unit 21 acquires the feature amount of the teacher time series data TD.

[0072] The clustering unit 22 clusters the feature amounts obtained by the feature amount obtaining unit 21. For example, k-means or TW-FINCH may be used as the clustering method.

[0073] The data selection unit 23 selects data from near the center of each cluster using the clustering results obtained by the clustering unit 22. For each cluster divided by feature of the teacher time-series data TD, the data selection unit 23 selects data from near the center of the cluster that represents each feature, and obtains the time of the selected data.

[0074] The label acquiring unit 24 acquires a label to be assigned to data at each time in the teacher time series data TD, which corresponds to the time obtained by the data selecting unit 23. For example, the label acquired by the label acquiring unit 24 may be assigned manually by a person to the data at each time in the teacher time series data TD.

[0075] The class matching unit 25, the class propagation unit 26, the pseudo label assignment unit 27, and the learning unit 28 have the same functions as the class matching unit 11, the class propagation unit 12, the pseudo label assignment unit 13, and the learning unit 14 described in exemplary embodiment 1, and therefore their explanations are omitted.

[0076] (Flow of learning method S20) The learning device 20 configured as above executes the learning method S20 according to this exemplary embodiment. The flow of the learning method S20 will be described with reference to Fig. 6. Fig. 6 is a flow chart showing the flow of the learning method S20. As shown in Fig. 6, the learning method S20 includes steps S201 to S208.

[0077] In the feature amount acquisition step S201, the feature amount acquisition unit 21 acquires the feature amount of the teacher time-series data TD.

[0078] In the clustering step S202, the clustering unit 22 clusters the feature amounts obtained in the feature amount obtaining step S201.

[0079] In the data selection step S203, the data selection unit 23 selects data from near the center of each cluster using the clustering results obtained in the clustering step S202.

[0080] In the label acquisition step S204, the label acquisition unit 24 acquires labels to be assigned to data at each time in the teacher time series data TD, which correspond to the time obtained in the data selection step S203. The labels acquired in the label acquisition step S204 are assigned to the data at each time in the teacher time series data TD, for example, manually by a person, and then the process proceeds to the class association step S205 and subsequent steps.

[0081] The class matching step S205, the class propagation step S206, the pseudo-labeling step S207, and the learning step S208 have the same processing as the class matching step S11, the class propagation step S12, the pseudo-labeling step S13, and the learning step S14 described in the exemplary embodiment 1, and therefore their explanations are omitted.

[0082] As described above, the learning device 20 and learning method S20 according to this exemplary embodiment make it possible to acquire data at each time point in the teacher time-series data TD that have different characteristics from each other in the teacher time-series data TD. By selecting in advance the data at each time point in the teacher time-series data TD acquired in this way as data to be labeled, it is possible to reduce the cost of searching for data to be labeled within the teacher time-series data TD.

[0083] [Software Implementation Example] Some or all of the functions of the learning devices 1, 10, and 20 (hereinafter referred to as each device) may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0084] In the latter case, each device is realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 7. Computer C includes at least one processor C1 and at least one memory C2. Memory C2 stores a program P for operating computer C as each device. In computer C, processor C1 reads and executes program P from memory C2, thereby realizing the functions of each device.

[0085] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0086] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0087] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0088] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0089] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects.

[0090] (Supplementary Note 1) A learning device that uses a plurality of teacher time series data to machine-train a machine learning model that infers into which class data at each time point in time series data is classified, wherein some data included in the plurality of teacher time series data are assigned labels indicating the classes, the device comprising: a class association unit that associates each teacher time series data with a class indicated by the label assigned to data included in the teacher time series data; a class propagation unit that associates at least one teacher time series data with at least a portion of the classes associated with other teacher time series data based on similarity between the teacher time series data; a pseudo label assignment unit that assigns, for each teacher time series data, a pseudo label indicating a class into which the machine learning model has classified data that is not assigned a label included in the teacher time series data; and a learning unit that trains the machine learning model by machine learning using the plurality of teacher time series data including data assigned the pseudo labels, The pseudo label assignment unit limits the pseudo labels to be assigned to data included in the teacher time-series data based on a class associated with each teacher time-series data.

[0091] a feature calculation unit that calculates features of each time series data unit for each of the teacher time series data; a similarity calculation unit that calculates the similarity between the teacher time series data using the features; and a constraint assignment unit that assigns a constraint that restricts the class of the pseudo label to the class of the label originally present in the teacher time series data or the class of the label in the teacher time series data obtained by assignment by the class propagation unit, wherein the learning unit calculates a loss using as input the label originally assigned in the teacher time series data, the pseudo label assigned to the teacher time series data, and a result of the inference, and updates parameters of the machine learning model using the loss as input.

[0092] (Supplementary Note 3) The learning device according to Supplementary Note 1 or 2, further comprising: a feature acquisition unit that acquires features of the teacher time series data; a clustering unit that clusters the features acquired by the feature acquisition unit; a data selection unit that selects data from near the center of each cluster using the clustering results acquired by the clustering unit; and a label acquisition unit that acquires the label to be assigned to data at each time point of the teacher time series data corresponding to the time point acquired by the data selection unit.

[0093] (Supplementary Note 4) The learning device according to Supplementary Note 2 or 3, wherein the feature is an output result of a pre-trained model, a color feature, or meta information.

[0094] (Supplementary Note 5) The learning device according to Supplementary Note 4, wherein the meta-information is an acquisition time of the time-series data or an acquisition location of the time-series data.

[0095] (Supplementary Note 6) A learning method for machine learning a machine learning model that infers into which class data at each time point of time series data is classified, using a plurality of teacher time series data, wherein some data included in the plurality of teacher time series data are assigned labels indicating the classes, the method comprising: a class association process for associating each teacher time series data with a class indicated by the label assigned to data included in the teacher time series data; a class propagation process for associating at least one teacher time series data with at least some of the classes associated with other teacher time series data based on the similarity between the teacher time series data; a pseudo-label assignment process for assigning, for each teacher time series data, a pseudo-label indicating the class into which the machine learning model has classified the data, to data included in the teacher time series data that is not assigned the label; and a learning process for machine learning the machine learning model using the plurality of teacher time series data including data assigned the pseudo-labels, The pseudo-labeling process limits the pseudo-labels to be assigned to data included in the training time-series data based on a class associated with each training time-series data.

[0096] (Supplementary Note 7) A learning program for causing a computer to perform machine learning on a machine learning model that infers into which class data at each time point in time series data is classified, using a plurality of teacher time series data, wherein some data included in the plurality of teacher time series data are assigned labels indicating the classes, the program causing a computer to execute the following: a class association process for associating each teacher time series data with a class indicated by the label assigned to data included in the teacher time series data; a class propagation process for associating at least one teacher time series data with at least some of the classes associated with other teacher time series data based on similarity between the teacher time series data; a pseudo-label assignment process for assigning, for each teacher time series data, a pseudo-label indicating a class into which the machine learning model has classified data to data included in the teacher time series data that is not assigned the label; and a learning process for training the machine learning model using the plurality of teacher time series data including data assigned the pseudo-labels, A learning program in which the pseudo-labeling process limits the pseudo-labels to be assigned to data included in each training time-series data based on a class associated with the training time-series data.

[0097] (Supplementary Note 8) A learning device including at least one processor, which uses a plurality of teacher time series data to machine-learn a machine learning model that infers into which class data at each time point of time series data is classified, wherein a label indicating the class is assigned to a portion of data included in the plurality of teacher time series data, and the processor executes the following steps: a class association process that associates each of the teacher time series data with a class indicated by the label assigned to data included in the teacher time series data; a class propagation process that associates at least one of the teacher time series data with at least a portion of the classes associated with other teacher time series data based on the similarity between the teacher time series data; a pseudo-label assignment process that assigns, for each of the teacher time series data, a pseudo-label indicating a class into which the machine learning model has classified the data, to data included in the teacher time series data that is not assigned the label; and a learning process that trains the machine learning model by machine learning using the plurality of teacher time series data including data assigned the pseudo-label, The pseudo-labeling process limits the pseudo-labels to be assigned to data included in the teacher time-series data based on a class associated with each teacher time-series data.

[0098] The learning device may further include a memory that stores a program for causing the processor to execute the class association process, the class propagation process, the pseudo-labeling process, and the learning process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0099] 1, 10, 20 Learning device 11, 25 Class matching unit 12, 26 Class propagation unit 13, 27 Pseudo label assignment unit 14, 28 Learning unit 15 Inference unit 16 Feature calculation unit 17 Similarity calculation unit 18 Constraint condition assignment unit 21 Feature acquisition unit 22 Clustering unit 23 Data selection unit 24 Label acquisition unit 110, 210 Control unit 120, 220 Storage unit C1 Processor C2 Memory

Claims

1. A learning device that performs machine learning to generate a machine learning model that infers into which class data at each time point in time-series data is classified, using a plurality of teacher time-series data, A label indicating the class is assigned to some data included in the plurality of training time-series data; a class association unit that associates each teacher time series data with a class indicated by the label assigned to data included in the teacher time series data; a class propagation unit that associates at least a part of classes associated with other teacher time series data with at least one teacher time series data based on the similarity between the teacher time series data; a pseudo-labeling unit that assigns a pseudo-label indicating a class into which the machine learning model has classified data to data that is not assigned a label and is included in the teacher time-series data for each teacher time-series data; a learning unit that uses the plurality of training time-series data sets including the data sets to which the pseudo-labels are assigned to learn the machine learning model; The pseudo label assignment unit limits the pseudo labels to be assigned to data included in the teacher time-series data based on a class associated with each teacher time-series data.

2. an inference unit that infers into which class data at each time point of the teacher time series data is classified; a feature calculation unit that calculates feature values ​​for each of the teacher time series data units; a similarity calculation unit that calculates the similarity between the training time-series data using the feature amount; a constraint condition assigning unit that assigns a constraint condition that restricts the class of the pseudo label to the class of the label originally present in the training time-series data or the class of the label in the training time-series data obtained by assigning the label by the class propagation unit, The learning unit Calculating a loss using the labels originally assigned to the training time-series data, the pseudo-labels assigned to the training time-series data, and the result of the inference as inputs; updating parameters of the machine learning model using the loss as an input; The learning device according to claim 1 .

3. a feature acquisition unit that acquires feature values ​​of the teacher time-series data; a clustering unit that clusters the feature amounts obtained by the feature amount acquisition unit; a data selection unit that selects data from near the center of each cluster using the clustering results obtained by the clustering unit; a label acquisition unit that acquires the label to be assigned to data at each time point of the teacher time-series data corresponding to the time point obtained by the data selection unit, The learning device according to claim 1 or 2.

4. The feature is an output result of a pre-trained model, a color feature, or meta information. The learning device according to claim 2 .

5. The meta information is the acquisition time of the time series data or the acquisition location of the time series data. The learning device according to claim 4 .

6. A learning method for machine learning a machine learning model that infers into which class data at each time point in time-series data is classified, using a plurality of training time-series data, comprising: A label indicating the class is assigned to some data included in the plurality of training time-series data; a class association process for associating each teacher time series data with a class indicated by the label assigned to the data included in the teacher time series data; a class propagation process for associating at least a part of classes associated with other teacher time series data with at least one teacher time series data based on the similarity between the teacher time series data; a pseudo-labeling process for assigning a pseudo-label indicating a class into which the machine learning model has classified each of the teacher time-series data to data that has not been assigned a label and is included in the teacher time-series data; a learning process for learning the machine learning model using the plurality of training time-series data sets including the data sets to which the pseudo-labels are assigned; The pseudo-labeling process limits the pseudo-labels to be assigned to data included in the training time-series data based on a class associated with each training time-series data.

7. On the computer, A learning program for performing machine learning on a machine learning model that infers into which class data at each time point in time-series data is classified using a plurality of training time-series data, the program comprising: A label indicating the class is assigned to some data included in the plurality of training time-series data; a class association process for associating each teacher time series data with a class indicated by the label assigned to the data included in the teacher time series data; a class propagation process for associating at least a part of classes associated with other teacher time series data with at least one teacher time series data based on the similarity between the teacher time series data; a pseudo-labeling process for assigning a pseudo-label indicating a class into which the machine learning model has classified each of the teacher time-series data to data that has not been assigned a label and is included in the teacher time-series data; a learning process for learning the machine learning model by using the plurality of training time-series data sets including the data sets to which the pseudo-labels have been assigned; A learning program in which the pseudo-labeling process limits the pseudo-labels to be assigned to data included in each training time-series data based on a class associated with the training time-series data.