Machine learning program, method, and device

Abstract

This machine learning device uses a video, which contains multiple frames and includes a label that indicates a type of action and that is assigned to a representative frame included in each section that has been divided into the type of action of a person in the video, to combine a first label with a second label and generate a combined label for each frame between a first representative frame assigned with the first label and a second representative frame assigned with the second label, and trains a machine learning model for inferring a label for each frame contained in an input video to maximize the probability that a label for each frame inferred by the machine learning model is either the first label or the second label included in a combined label generated for each frame.

Description

Machine learning program, method, and device

[0001] The disclosed technology relates to a machine learning program, a machine learning method, and a machine learning device.

[0002] Human movements in video are estimated using machine learning models. To train such machine learning models, videos with correct labels indicating the type (class) of movement are used as training data. The ideal case of training data is one in which correct labels are assigned to each frame (hereinafter referred to as "fully annotated"). However, preparing fully annotated training data presents two challenges. First, assigning correct labels to each frame requires enormous work. Second, the temporal boundaries between movement types may be unclear, and different annotators may assign different labels to frames near the boundaries. This may result in biased data.

[0003] Therefore, a method called timestamp annotation has been proposed, which labels only one frame among multiple frames included in a section showing a single action, instead of labeling all frames. This method reduces the labor cost of labeling compared to full annotation. Furthermore, this method allows annotators to select reliable timestamps for labeling, thereby reducing label inconsistencies at temporal boundaries.

[0004] Z. Li, Y. Abu Farha, and J. Gall, "Temporal action segmentation from timestamp supervision," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8365-8374.

[0005] However, there is a problem in that machine learning models trained with timestamp annotation training data are less accurate than machine learning models trained with full annotation training data.

[0006] In one aspect, the disclosed technology aims to improve the accuracy of machine learning models for estimating the movements of people in video without full annotation.

[0007] In one aspect, the disclosed technology uses a video including a plurality of frames, in which a label indicating a type of a person's movement is assigned to a representative frame included in each section divided into sections for each type of the person's movement. The disclosed technology generates a combined label by combining the first label and the second label for each frame between a first representative frame assigned a first label and a second representative frame assigned a second label in the video. The disclosed technology then trains the machine learning model to maximize the probability that the label of each frame estimated by the machine learning model is the first label or the second label included in the combined label generated for each frame. The machine learning model estimates the label of each frame included in the input video.

[0008] One aspect of this is that it has the effect of improving the accuracy of machine learning models for estimating the movements of people in video without performing full annotation.

[0009] 1 is a functional block diagram of a machine learning device. FIG. 2 is a schematic diagram showing an example of training video. FIG. 3 is a diagram for explaining generation of combined labels. FIG. 4 is a diagram for explaining training of a machine learning model using combined labels. FIG. 5 is a block diagram showing the schematic configuration of a computer functioning as a machine learning device. FIG. 6 is a flowchart showing an example of machine learning processing. FIG. 7 is a flowchart showing an example of estimation processing. FIG. 8 is a diagram for explaining a comparison of estimation results between the present method and comparative method 1. FIG. 9 is a diagram for explaining a comparison of estimation results between the present method and comparative method 2. FIG. 10 is a diagram for explaining an example application of the machine learning device according to the present embodiment to a scoring system for gymnastics.

[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0011] As shown in FIG. 1, a training video is input to a machine learning device 10 according to this embodiment when training a machine learning model 20, and an estimation target video is input to the machine learning device 10 when estimating a movement.

[0012] In training videos, labels indicating the type (class) of action are assigned to some frames by timestamp annotation. Here, the labels assigned by timestamp annotation will be described in comparison with full annotation. FIG. 2 is a diagram schematically illustrating an example of training video. The diagram in the upper part of FIG. 2 is a schematic diagram of some frames included in the video arranged in chronological order from left to right, the diagram in the middle part is a schematic diagram of labels assigned by full annotation, and the diagram in the lower part is a schematic diagram of labels assigned by timestamp annotation. In the schematic diagrams of the labels in the middle and lower parts, the width shown in the leftmost part of the diagram in the middle part corresponds to one frame, and differences in the labels of each frame are indicated by different hatching.

[0013] In full annotation, all frames in a video are labeled. In Fig. 2, the same label (c １ , c ２ , c ３ , and c ４ ) are labeled as blocks. As mentioned above, full annotation has the drawbacks of being extremely costly to label, and of being ambiguous about the time boundaries between action types (the dashed lines in the middle of Figure 2), which can lead to inconsistent labels between annotators.

[0014] On the other hand, with timestamp annotation, a label is assigned to only one frame out of multiple frames included in a section showing a single action. This reduces the cost of labeling and eliminates label inconsistencies at temporal boundaries. When training a machine learning model using training videos labeled with timestamp annotation, pseudo labels (shown by the two-dot chain line in the lower part of Figure 2) are generated for frames other than those labeled correctly. These pseudo labels are less reliable as correct answers because they represent all possible labels that the machine learning label can output. As a result, the estimation accuracy of the trained machine learning model is inferior to that of a machine learning model trained with fully annotated training videos. Hereinafter, training a machine learning model using training videos labeled with timestamp annotation is referred to as "timestamp semi-supervised learning."

[0015] Therefore, in this embodiment, a machine learning model is trained by generating combined labels (details of which will be described later) that are more reliable than pseudo labels generated during timestamp semi-supervised learning. The machine learning device 10 according to this embodiment will be described in detail below.

[0016] 1 , the machine learning device 10 functionally includes a machine learning unit 12 and an estimation unit 18. The machine learning unit 12 further includes a generation unit 14 and a training unit 16. A machine learning model 20 is stored in a predetermined storage area of ​​the machine learning device 10. The machine learning model 20 is a model that estimates the label of each frame included in an input video, and is, for example, a model such as a deep neural network.

[0017] The generation unit 14 acquires training videos input to the machine learning device 10. The generation unit 14 generates a combined label by combining the first label and the second label for each frame between a first representative frame assigned with a first label and a second representative frame assigned with a second label in the acquired training videos.

[0018] Specifically, the generation unit 14 assigns a first label to each frame from the first representative frame to the second representative frame up to the frame immediately preceding the second representative frame. The generation unit 14 also assigns a second label to each frame from the second representative frame to the first representative frame up to the frame immediately preceding the first representative frame. The generation unit 14 then generates a combined label by combining the multiple labels assigned to each frame. The representative frame is a frame to which a label is assigned using a timestamp annotation.

[0019] For example, as shown in A of FIG. 3, the generation unit 14 generates a label c １ The next frame in the chronological order from the frame with label c １ The assignment of the label c ２ 3B, the generation unit 14 repeats the process up to the frame immediately before the frame to which the label c １ From the frame to which label c is assigned, the previous frame is assigned label c in reverse chronological order. １ This process is repeated until the first frame. As a result, as shown in D of FIG. 3, the label c ２ Label each frame up to the frame immediately before the frame to which label c is assigned. １ will be granted.

[0020] Similarly, as shown in FIG. 3E, the generation unit 14 generates a label c ２ The next frame in the chronological order from the frame with label c ２ The assignment of the label c ３ 3F, the generation unit 14 repeats the process up to the frame immediately before the frame (not shown) to which the label c ２ From the frame to which label c is assigned, the previous frame is assigned label c in reverse chronological order. ２ The assignment of the label c １ This is repeated until the frame immediately following the frame to which label c is assigned. １ The frame immediately following the frame with label c ３Label each frame up to the frame immediately before the frame to which label c is assigned. ２ will be granted.

[0021] The generation unit 14 performs the above process on all frames to which labels are assigned using timestamp annotations, i.e., all representative frames. Then, for example, the generation unit 14 generates a frame labeled c １ and C ２ The combined label c １ ∪c ２ Generate.

[0022] The training unit 16 trains the machine learning model 20 so as to maximize the probability that the label of each frame is the first label or the second label included in the combined label generated for that frame. In this embodiment, the machine learning model 20 estimates the probability that the label of each frame is each of multiple labels indicating the type of action, with a value between 0 and 1. Specifically, the training unit 16 trains the machine learning model 20 so as to minimize a loss function that becomes smaller as the sum of the probability that the label of a frame for which a combined label has been generated is the first label and the probability that it is the second label approaches 1.

[0023] More specifically, the number of frames in the training video is N ｆｒａｍｅ , the number of label types is N Ｃ Then, the output Y (real number) of the machine learning model 20 is N ｆｒａｍｅ ×N ｃ The output of one neuron of the machine learning model 20 is expressed as a matrix of y ｉ Then, each element of the matrix Y is Y[i,f]=p(y ｉ，ｆ ), that is, the label of frame f is c ｉ The probability that p(y ｉ，ｆ ) is generally formulated by the following equation (1):

[0024]

[0025] The training unit 16 uses, for example, the mean square error to estimate the probability p(y ｉ，ｆ) and the true probability of the combined label. ａｕ is defined as in the following equation (2).

[0026]

[0027] N Ｃ ｐｏｓ is the label c included in the combined label ｉ The numerator in the parentheses on the right side of equation (2) is the number of labels c included in the bond label. ｉ The probability p(y ｉ，ｆ Since the denominator in the parentheses on the right side of equation (2) is 1, the closer the numerator is to 1, the smaller the loss function L ａｕ becomes smaller.

[0028] For example, as shown in FIG. 4, a representative frame is labeled c １ , c ２ , c ３ , and c ４ 4 , the machine learning model 20 is trained using training video including frames with labels c . First, for comparison, the case where timestamp semi-supervised learning is performed using this training video is described. ３ In the case of a representative frame to which , the probability estimated by the machine learning model 20 is p(c １ ) = 0, p(c ２ ) = 0, p(c ３ ) = 1, and p(c ４ ) = 0. However, for frames that are not representative frames, such as frames K and M in FIG. １ ), p(c ２ ), p(c ３ ), and p(c ４ ) should be 1 and which should be 0. As a result, training of the machine learning model 20 will depend on unreliable pseudo-labels, resulting in reduced estimation accuracy.

[0029] On the other hand, in this embodiment, the combined label c １ ∪c ２For a frame in which p(c １ ∪c ２ ) = 1, and p(c ３ ∪c ４ ) = 0. Also, the combined label c ３ ∪c ４ For a frame in which p(c １ ∪c ２ ) = 0, and p(c ３ ∪c ４ ) = 1. In this way, in this embodiment, a loss function is used such that the sum of the probabilities of labels included in the combined label approaches 1 and the sum of the probabilities of labels not included in the combined label approaches 0. This makes it possible to generate highly reliable combined labels for frames other than the representative frame and train the machine learning model 20.

[0030] The training unit 16 stores the trained machine learning model 20 in a predetermined storage area of ​​the machine learning device 10 .

[0031] The estimation unit 18 acquires the estimation target video input to the machine learning device 10. The estimation unit 18 inputs the estimation target video to a trained machine learning model 20, and estimates the motion indicated by each frame included in the estimation target video. Specifically, the estimation unit 18 calculates p(c ｉ，ｆ ) is maximized by the label c ｉ is estimated as the motion of frame f and output as the estimation result.

[0032] The machine learning device 10 may be realized by, for example, a computer 40 shown in FIG. 5 . The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 serving as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, I / O device 45, R / W device 46, and communication I / F 47 are connected to one another via a bus 48.

[0033] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 44, which serves as a storage medium, stores a machine learning program 50 for causing the computer 40 to function as the machine learning device 10. The machine learning program 50 includes a generation process control instruction 54, a training process control instruction 56, and an estimation process control instruction 58. The storage device 44 also includes an information storage area 60 in which information constituting the machine learning model 20 is stored.

[0034] The CPU 41 reads the machine learning program 50 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions of the machine learning program 50. The CPU 41 operates as the generation unit 14 shown in FIG. 1 by executing the generation process control instruction 54. The CPU 41 also operates as the training unit 16 shown in FIG. 1 by executing the training process control instruction 56. The CPU 41 also operates as the estimation unit 18 shown in FIG. 1 by executing the estimation process control instruction 58. The CPU 41 also reads information from the information storage area 60 and loads the machine learning model 20 into the memory 43. As a result, the computer 40 that has executed the machine learning program 50 functions as the machine learning device 10. The CPU 41 that executes the program is hardware. A portion of the program may also be executed by the GPU 42.

[0035] The functions realized by the machine learning program 50 may be realized, for example, by a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0036] Next, the operation of the machine learning device 10 according to this embodiment will be described. When training video is input to the machine learning device 10 and an instruction to train the machine learning model 20 is given, the machine learning device 10 executes the machine learning process shown in Fig. 6. When estimation target video is input to the machine learning device 10 and an instruction to estimate a movement is given, the machine learning device 10 executes the estimation process shown in Fig. 7. Note that the machine learning process is an example of a machine learning method of the disclosed technology.

[0037] First, the machine learning process shown in FIG. 6 will be described.

[0038] In step S10, the generation unit 14 acquires training video input to the machine learning device 10. Next, in step S12, the generation unit 14 assigns the label of the representative frame assigned by the timestamp annotation to each frame up to the frame immediately preceding the adjacent representative frame in chronological order. The generation unit 14 also assigns the label of the representative frame assigned by the timestamp annotation to each frame up to the frame immediately following the adjacent representative frame in reverse chronological order. Then, the generation unit 14 generates a combined label for each frame by combining the multiple labels assigned to that frame.

[0039] Next, in step S14, the training unit 16 trains the machine learning model 20 so as to maximize the probability that the label of each frame is the first label or the second label included in the combined label generated for that frame. Then, the training unit 16 stores the trained machine learning model 20 in a predetermined storage area of ​​the machine learning device 10, and the machine learning process ends.

[0040] Next, the estimation process shown in FIG. 7 will be described.

[0041] In step S20, the estimation unit 18 acquires the estimation target video input to the machine learning device 10. Next, in step S22, the estimation unit 18 inputs the estimation target video to the trained machine learning model 20, estimates the motion indicated by each frame included in the estimation target video, and outputs the estimation result, thereby completing the estimation process.

[0042] As described above, the machine learning device according to this embodiment uses, as training video, video including a plurality of frames, in which representative frames included in each section are divided into sections for each type of human movement, and labels indicating the type of movement are assigned to the representative frames. The machine learning device generates a combined label by combining the first label and the second label for each frame between a first representative frame assigned a first label and a second representative frame assigned a second label in the training video. The machine learning device then trains the machine learning model to maximize the probability that the label of each frame estimated by the machine learning model is the first label or the second label included in the combined label generated for each frame. This improves the accuracy of the machine learning model for estimating the movements of a person in video without performing full annotation.

[0043] FIG. 8 shows the comparison results for each of videos 1 to 3 between the correct labels, the labels estimated by comparative method 1, and the labels estimated by the method of this embodiment (hereinafter referred to as "this method"). In FIG. 8, as in the above-described FIGS. 2 to 4, differences in labels are represented by different hatching. The same applies to FIG. 9, which will be described later. Comparative method 1 is a method of training a machine learning model using training videos labeled by full annotation. The estimation results of this method are very close to the correct answer, and an estimation accuracy that can be said to be within an acceptable range for use in applications is obtained.

[0044] 9 shows the comparison results for each of videos 1 to 3, between the correct labels, the labels estimated by comparative method 2, and the labels estimated by our method. Comparative method 2 is timestamp semi-supervised learning. In particular, it can be seen that our method has improved estimation accuracy compared to comparative method 2, especially in the areas surrounded by the bold lines in FIG. 9.

[0045] In the above embodiment, the case where the motion indicated by the label with the highest probability is output as the estimation result has been described, but this is not limiting. The estimation result may be the probability that the label indicating the motion of each frame, which is the output of the machine learning model, is one of multiple labels, i.e., Y[i, f].

[0046] Furthermore, in the above embodiment, the machine learning unit and the estimation unit are configured as one computer, but the machine learning unit and the estimation unit may be configured as separate computers.

[0047] The above-described embodiment can also be applied to, for example, human-robot interactions. Specifically, a robot captures human motions with a camera and estimates the human motions from the captured video using a machine learning model trained as in the above-described embodiment. The robot is then controlled to assist or imitate the human's actions based on the estimated motions.

[0048] The above embodiment can also be applied to a scoring system for gymnastics, for example. An example of the processing of the scoring system for gymnastics will now be outlined with reference to FIG.

[0049] When a set of multi-view images of an object captured from multiple different viewpoints is input, the scoring system detects a person's region from each image included in the multi-view images. The scoring system tracks the person by matching regions representing the same person in multiple frames from a single viewpoint in the time-series multi-view images. The scoring system also determines whether the person represented by the detected region is a player or a non-player, identifies the region representing the player, and matches the tracked player between the multiple viewpoints, i.e., between the images. The scoring system recognizes the player's two-dimensional skeletal information from each of the tracked images using a recognition model or the like. The scoring system estimates three-dimensional skeletal information from the two-dimensional skeletal information using camera parameters. The scoring system then performs post-processing, such as smoothing, on the time-series three-dimensional skeletal information to estimate the phases (breaks) of the performance, and then recognizes the technique. A machine learning model trained by the machine learning device according to the above embodiment can be applied to this technique recognition.

[0050] The application of the disclosed technology is not limited to the above-mentioned human-robot interaction, gymnastics scoring system, etc., but can also be applied to general motion recognition applications.

[0051] In the above embodiment, the machine learning program is stored (installed) in advance in a storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0052] 10 Machine learning device 12 Machine learning unit 14 Generation unit 16 Training unit 18 Estimation unit 20 Machine learning model 30 Estimation unit 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / output device 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50 Machine learning program 54 Generation process control command 56 Training process control command 58 Estimation process control command 60 Information storage area

Claims

1. In a video including a plurality of frames, a label indicating a type of a person's movement is assigned to a representative frame included in each section divided into sections, the representative frame including the plurality of frames being divided into sections divided into sections each including a type of the person's movement, and for each frame between a first representative frame assigned with a first label and a second representative frame assigned with a second label, a combined label is generated by combining the first label and the second label; training a machine learning model that estimates a label of each frame included in an input video so as to maximize the probability that the label of each frame estimated by the machine learning model is the first label or the second label included in the combined label generated for each frame; A machine learning program that allows a computer to perform processes including:

2. the machine learning model estimates, as a value between 0 and 1, a probability that the label of each frame is one of a plurality of labels indicating the type of the action; The process of training the machine learning model includes minimizing a loss function that decreases as the sum of the probability that the label of the frame for which the combined label is generated is the first label and the probability that the label is the second label approaches 1. The machine learning program according to claim 1 .

3. 3. The machine learning program according to claim 1, wherein the process of generating the combined label includes assigning the first label to each frame from the first representative frame to the second representative frame up to the frame immediately preceding the second representative frame, assigning the second label to each frame from the second representative frame to the first representative frame up to the frame immediately preceding the first representative frame, and combining the multiple labels assigned to each frame to generate the combined label.

4. 3. The machine learning program according to claim 2, for causing the computer to execute a process including, when a video for which labels are to be estimated is input to the trained machine learning model, outputting, as the label of each frame, the label estimated by the machine learning model for each frame of the video for which labels are to be estimated, which has the highest probability of being each of the plurality of labels.

5. In a video including a plurality of frames, a label indicating a type of a person's movement is assigned to a representative frame included in each section divided into sections, the representative frame including the plurality of frames being divided into sections divided into sections each including a type of the person's movement, and for each frame between a first representative frame assigned with a first label and a second representative frame assigned with a second label, a combined label is generated by combining the first label and the second label; training a machine learning model that estimates a label of each frame included in an input video so as to maximize the probability that the label of each frame estimated by the machine learning model is the first label or the second label included in the combined label generated for each frame; A machine learning method in which a computer performs a process including:

6. a generation unit that generates a combined label by combining the first label and the second label for each frame between a first representative frame to which a first label is assigned and a second representative frame to which a second label is assigned, in a video including a plurality of frames, the video being divided into sections each divided into sections each corresponding to a type of human motion and each of the sections being assigned a label indicating the type of the motion; a training unit that trains a machine learning model that estimates a label of each frame included in an input video so as to maximize the probability that the label of each frame estimated by the machine learning model is the first label or the second label included in the combined label generated for each frame; Machine learning devices including: