Machine learning program, machine learning method, and information processing device

Abstract

This information processing device acquires video in which a work performed by a person was imaged. The information processing device receives, for each time-series section of the acquired video, a label indicating a work element of the person. The information processing device performs, on the basis of the received label, training process for training the transition probability of a feature amount per unit time forming the work element. When a label of a particular type has been imparted to the entirety of, or a portion of, the work element, the training process performed by the information processing device changes, on the basis of the type of the imparted label, the transition probability of the feature amount for the section corresponding to the label.

Description

Machine learning program, machine learning method, and information processing device

[0001] The present invention relates to a machine learning program and the like.

[0002] Advances in deep learning technology have made it possible to accurately recognize postures from images of people taken with standard RGB (Red, Green, Blue) cameras, and various research and development efforts are underway to use this recognition information to estimate human behavior.

[0003] For example, there is a conventional technology that uses a camera to film a person working, identifies multiple unit actions in chronological order based on the work video, and compares the sequence of the identified unit actions with a model to infer the multiple elemental tasks contained in the work video.

[0004] Since the sequence of unit actions during an element task does not always match perfectly and there is variation, conventional technology uses a model that expresses the sequence of unit actions during an element task as probabilistic transitions (transition probabilities) between unit actions. In conventional technology, the transition probabilities of the model are trained (learned) based on training data that defines the relationship between the patterns of the sequence of unit actions actually observed in work footage and the labels of element tasks.

[0005] FIG. 15 is a diagram for explaining the prior art. For convenience, the prior art device is referred to as the "prior art device." The prior art device estimates unit actions included in a work video in time series based on the work video 5. In the example shown in FIG. 15, unit actions m ２ , m ５ , m ６ , m ８ , m ６ , m ９ , m ５ The unit actions are estimated in the order of , .... For example, a conventional device estimates each unit action using a model trained with unsupervised data.

[0006] The conventional device estimates the sequence of unit movements as described above, while comparing it with multiple models corresponding to each element task, and sequentially identifies the element tasks. The example shown in Fig. 15 shows a model 20A corresponding to element task A and a model 20B corresponding to element task B. Models 20A and 20B are HMMs (Hidden Markov Models).

[0007] Models 20A and 20B have unit operations m ｎ (n is a natural number). Each state node is connected to a predetermined state node by an edge. A transition probability learned from teacher data is set for each edge. In FIG. 15, state nodes corresponding to unit actions of n=4 or more are not shown.

[0008] In the prior art, the state nodes of the models 20A and 20B are transitioned based on the estimation result of the unit actions, and the element tasks corresponding to the sequence of the unit actions are identified in order based on the transition probability of the edge that has actually transitioned and various constraints. The various constraints include the work time of the element tasks and the order of the element tasks. In the example shown in FIG. 15, as a result of the comparison, the unit action m ２， m ５ , m ６ The sequence of these is identified as element work A, and the unit action m ８， m ６ , m ９ , m ５ is identified as element operation B.

[0009] Japanese Patent Application Laid-Open No. 2021-189892

[0010] However, the above-mentioned conventional techniques have a problem in that they are unable to improve the accuracy of identifying elemental tasks.

[0011] In one aspect, the present invention aims to provide a machine learning program, a machine learning method, and an information processing device that can improve the accuracy of identifying element tasks.

[0012] In the first proposal, a computer is caused to execute the following process. The computer acquires video footage of a person working. The computer accepts labels indicating the person's elemental work for each time-series section of the acquired video. The computer executes a learning process to learn the transition probabilities of the features per unit time that make up the elemental work, based on the accepted labels. When a specific type of label is assigned to all or part of the elemental work, the learning process executed by the computer changes the transition probabilities of the features in the section corresponding to the label, based on the type of the assigned label.

[0013] This can improve the accuracy of identifying a person's activity.

[0014] FIG. 1 is a diagram (1) illustrating a problem with the conventional technology. FIG. 2 is a diagram (2) illustrating a problem with the conventional technology. FIG. 3 is a diagram illustrating an example of a system according to the present embodiment. FIG. 4 is a diagram (1) illustrating processing by an information processing device according to the present embodiment. FIG. 5 is a diagram (2) illustrating processing by an information processing device according to the present embodiment. FIG. 6 is a diagram illustrating an effect of the information processing device according to the present embodiment. FIG. 7 is a functional block diagram illustrating a configuration of an information processing device according to the present embodiment. FIG. 8 is a diagram illustrating processing by an observation probability learning unit. FIG. 9 is a diagram illustrating an example of processing by a transition probability learning unit. FIG. 10 is a diagram illustrating an example of a machine learning model. FIG. 11 is a flowchart (1) illustrating a processing procedure of an information processing device according to the present embodiment. FIG. 12 is a flowchart (2) illustrating a processing procedure of an information processing device according to the present embodiment. FIG. 13 is a flowchart (3) illustrating a processing procedure of an information processing device according to the present embodiment. FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment. FIG. 15 is a diagram illustrating the prior art.

[0015] Hereinafter, embodiments of the machine learning program, the machine learning method, and the information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0016] Before describing this embodiment, the problems of the conventional technology will be specifically described. FIGS. 1 and 2 are diagrams for explaining the problems of the conventional technology. First, FIG. 1 will be described. For example, a conventional device uses training data 30 to perform learning (machine learning) of models 20A, 20B, 20C, and 20D for each element task. The training data 30 includes a sequence of multiple unit actions corresponding to element task A, a sequence of multiple unit actions corresponding to element task B, a sequence of multiple unit actions corresponding to element task C, and a sequence of multiple unit actions corresponding to element task D. In this embodiment, "performing model learning" based on the training data will be referred to as "training a model" as appropriate.

[0017] Of the unit actions in the time series included in element work A, the time from the first unit action to the last unit action is defined as section T1-1. Of the unit actions in the time series included in element work B, the time from the first unit action to the last unit action is defined as section T1-2. Of the unit actions in the time series included in element work C, the time from the first unit action to the last unit action is defined as section T1-3. Of the unit actions in the time series included in element work D, the time from the first unit action to the last unit action is defined as section T1-4.

[0018] The conventional device trains the transition probability of a model 20A for an element task A using a sequence of multiple unit actions corresponding to element task A in the teacher data 30. The conventional device trains the transition probability of a model 20B for an element task B using a sequence of multiple unit actions corresponding to element task B in the teacher data 30. The conventional device trains the transition probability of a model 20C for an element task C using a sequence of multiple unit actions corresponding to element task C in the teacher data 30. The conventional device trains the transition probability of a model 20D for an element task D using a sequence of multiple unit actions corresponding to element task D in the teacher data 30.

[0019] Here, the person performing the work eliminates unnecessary actions in the process of performing the same work and performs work improvement. For example, even if the order of the unit actions included in each element work is correct when a model of each element work is learned using the training data 30, subsequent improvements may result in the inclusion of unnecessary unit actions in the element work. Here, we will proceed to the explanation of Figure 2, assuming that element work B in the training data 30 includes a unnecessary unit action.

[0020] Next, we will explain Fig. 2. The conventional device estimates the sequence of unit actions included in the target work video, and by comparing the estimated result with the trained models 20A to 20D described in Fig. 1, we obtain a detection result 32. On the other hand, the correct answer to the detection result of the element work included in the target work video is defined as a detection result 31.

[0021] The detection result 32 includes, from the top, element work A, element work B, element work C, and element work D. In the detection result 32, of the unit actions in the time series included in element work A, the time from the first unit action to the last unit action is defined as section T2-1. Of the unit actions in the time series included in element work B, the time from the first unit action to the last unit action is defined as section T2-2. Of the unit actions in the time series included in element work C, the time from the first unit action to the last unit action is defined as section T2-3. Of the unit actions in the time series included in element work D, the time from the first unit action to the last unit action is defined as section T2-4.

[0022] The correct detection result 31 includes, from the top, element work A, element work B, element work C, and element work D. In the detection result 31, of the unit actions in the time series included in element work A, the time from the first unit action to the last unit action is defined as section T3-1. Of the unit actions in the time series included in element work B, the time from the first unit action to the last unit action is defined as section T3-2. Of the unit actions in the time series included in element work C, the time from the first unit action to the last unit action is defined as section T3-3. Of the unit actions in the time series included in element work D, the time from the first unit action to the last unit action is defined as section T3-4.

[0023] For example, in the correct detection result 31, unnecessary actions have been removed from element work B through kaizen, and the section T3-2 of element work B in detection result 31 is shorter than the section T1-2 of element work B in training data 30.

[0024] Comparing the correct detection result 31 with the detection result 32, the order of the elemental tasks is the same, but the length of the section of elemental task B is significantly different, and it can be said that the accuracy of identifying the elemental tasks is poor. This is because the model was trained using elemental tasks B, which are included in the training data 30 and include unnecessary actions, and therefore elemental tasks B in the detection result 32 also include unnecessary actions.

[0025] Next, an example of the configuration of a system according to this embodiment will be described. Fig. 3 is a diagram showing an example of a system according to this embodiment. As shown in Fig. 3, this system includes a camera 15 and an information processing device 100. The camera 15 and the information processing device 100 are connected to each other via a network 16.

[0026] The camera 15 is an RGB camera or the like. The camera 15 captures video of the worker 14 working in a factory or the like, and transmits the captured video data to the information processing device 100. In the following description, the video data captured by the camera 15 will be referred to as "work video data." The work video data includes time-series frames (still images).

[0027] Next, the processing of the information processing device 100 will be described. FIGS. 4 and 5 are diagrams for explaining the processing of the information processing device 100 according to this embodiment. First, FIG. 4 will be described. The information processing device 100 performs the following processing based on work video data 40. The work video data 40 is data obtained by capturing video of people working in a factory or the like using an RGB camera, and includes time-series frames (still images). The information processing device 100 calculates time-series feature vectors for each predetermined window width based on the work video data 40.

[0028] The information processing device 100 estimates time-series unit movements by inputting time-series feature vectors into a first model 80. The first model 80 is a model that estimates unit movements from feature vectors, and is assumed to have been trained in advance using unsupervised data. For example, unit movements include "raising an arm," "lowering an arm," and "stretching an arm forward." A unit movement corresponds to a "feature amount."

[0029] In the example shown in FIG. 4, when a time-series feature vector is input, the first model 80 calculates a unit motion m ２ , m ５ , m ６ , m ８ , m ６ , m ９ , m ５ , ... are output in order.

[0030] In the learning phase for training the second model, a label identifying a person's elemental work is set for each time-series section in the work video data 40. For example, the label "elemental work A" is set for section T10-1, and the label "elemental work B" is set for section T10-2.

[0031] The information processing device 100 defines the unit action estimated from each feature vector corresponding to the section T10-1 of the work video data 40 as “unit action m ２ , m ５ , m ６ , m ８ ", then the unit action m ２ , m ５ , m ６ , m ８ The sequence of steps is associated with the label "element work A."

[0032] The information processing device 100 defines the unit action estimated from each feature vector corresponding to the section T10-2 of the work video data 40 as “unit action m ６ , m ９ , m ５ ", then the unit action m ６ , m ９ , m ５ The sequence of steps is associated with the label "element work B."

[0033] The information processing device 100 has a plurality of second models for identifying each element work. As an example, FIG. 4 shows a second model 90A corresponding to element work A and a second model 90B corresponding to element work B. The second models 90A and 90B each have a unit motion m ｎ (n is a natural number). Each state node is connected to a predetermined state node by an edge. An initial value of the transition probability is set for each edge. In FIG. 3, state nodes corresponding to unit actions of n=4 or more are not shown.

[0034] The information processing device 100 calculates the sequence of unit actions "m ２ , m ５ , m ６ , m ８ Based on this, the transition probabilities set for each edge of the second model 90A are trained.

[0035] The information processing device 100 calculates the sequence of unit actions "m ６ , m ９ , m ５ Based on this, the transition probabilities set for each edge of the second model 90B are trained.

[0036] The information processing device 100 repeatedly performs the above process on other work video data to train the transition probabilities of each second model.

[0037] Here, the information processing device 100 executes the process shown in FIG. 4 to train the transition probabilities of each second model, and then, when it receives an instruction that some unit actions in a sequence of unit actions corresponding to elemental work are wasteful due to work improvement, it executes the process shown in FIG. 5.

[0038] 5. When the information processing device 100 receives the designation of a wasteful unit action among a sequence of unit actions included in a certain element work, it sets a "waste label" on the designated wasteful unit action.

[0039] For example, the information processing device 100 may generate a sequence of unit actions "m ２ , m ５ , m ６ , m８ " Among them, the unit action "m ５ "," "m ６ " receives information that the unit action is useless, and the unit action "m ５ "," "m ６ In the explanation of FIG. 5, the unit action "m ５ "," "m ６ " is simply expressed as a unit action m ５ , m ６ It is written as follows.

[0040] The information processing device 100 selects the unit action m among the edges of each state node of the second model 90A. ２ From the state node of ５ The transition probability of the edge to the state node of unit action m ５ From the state node of ６ The information processing device 100 reduces the transition probability of the edge to the state node of unit action m ６ From the state node of ８ Decrease the transition probability of the edge to the state node of

[0041] The information processing device 100 calculates the sequence of unit operations "m ２ , m ５ , m ６ , m ８ ", then the unit action "m ５ , m ６ A sequence of unit actions, excluding "m ２ , m ８ " may be used to re-learn the transition probabilities set for each edge of the second model 90A.

[0042] As described above, when an unnecessary unit action is specified in the sequence of unit actions corresponding to an element work, the information processing device 100 according to this embodiment can improve the accuracy of identifying a person's element work by updating the transition probability of the second model of the corresponding element work.

[0043] 6 is a diagram illustrating the effects of the information processing device according to this embodiment. For example, the information processing device 100 according to this embodiment trains the transition probabilities of the second model corresponding to each element task based on the sequence of unit actions included in each of element tasks A to D set in the training data 30. Furthermore, when the information processing device 100 receives an instruction that a part or all of the unit actions included in element task B are wasteful, it updates the transition probabilities of the second model for the corresponding element task.

[0044] The information processing device 100 estimates the sequence of unit actions from the feature vectors of the target task video, and compares the estimated result with a second model corresponding to each element task to obtain a detection result 33. The correct detection result 31 and the detection result 32 of the prior art are explained in the same manner as in FIG. 2.

[0045] The detection result 33 includes, from the top, element work A, element work B, element work C, and element work D. In the detection result 33, of the unit actions in the time series included in element work A, the time from the first unit action to the last unit action is defined as section T4-1. Of the unit actions in the time series included in element work B, the time from the first unit action to the last unit action is defined as section T4-2. Of the unit actions in the time series included in element work C, the time from the first unit action to the last unit action is defined as section T4-3. Of the unit actions in the time series included in element work D, the time from the first unit action to the last unit action is defined as section T4-4.

[0046] Comparing the correct detection result 31 with the detection result 33, the order of the elemental tasks is the same, and the lengths of the sections of each elemental task are also approximately the same. In other words, the information processing device 100 according to this embodiment can improve the accuracy of identifying a person's task compared to the conventional technology.

[0047] Next, an example of the configuration of an information processing device 100 that executes the processes described in Figures 4 and 5 will be described. Figure 7 is a functional block diagram showing the configuration of an information processing device according to this embodiment. As shown in Figure 7, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0048] The communication unit 110 executes data communication with the camera 15, external devices, etc. via the network 16. The control unit 150, which will be described later, exchanges data with external devices via the communication unit 110.

[0049] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0050] The display unit 130 is a display device that displays information output from the control unit 150 .

[0051] The storage unit 140 includes a model table 141, an unsupervised data table 142, a supervised data table 143, an estimation result table 144, and a video buffer 145. The storage unit 140 corresponds to a storage device such as a memory.

[0052] 4 and a plurality of second models 90A and 90B. The second model 90A is a model corresponding to element work A. The second model 90B is a model corresponding to element work B. The model table 141 may further include second models corresponding to other element works.

[0053] The unsupervised data table 142 includes a plurality of unsupervised data. For example, the unsupervised data may be working video data divided into predetermined time intervals. The unsupervised data table 142 is used when training the first model 80.

[0054] The supervised data table 143 is a table that holds a sequence of unit actions. Each unit action held in the supervised data table 143 is assigned a label that identifies the element work. Furthermore, when an instruction for a wasteful unit action among the sequence of unit actions included in an element work is received through kaizen, a "waste label" is assigned to the label of the wasteful unit action. The supervised data table 143 is used when training each second model.

[0055] The estimation result table 144 is a table that holds the estimation results by the estimation unit 156, which will be described later.

[0056] The video buffer 145 is a buffer that stores the work video data acquired from the camera 15 .

[0057] The control unit 150 includes an acquisition unit 151, a reception unit 152, an extraction unit 153, an observation probability learning unit 154, a transition probability learning unit 155, an estimation unit 156, and a generation unit 157. The control unit 150 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0058] The acquisition unit 151 acquires work video data from the camera 15 and stores the acquired work video data in the video buffer 145 .

[0059] The acquiring unit 151 may also acquire the data of the unsupervised data table 142 and the data of the supervised data table 143 from an external device (not shown) or the like via the network 16. The acquiring unit 151 stores the acquired data of the unsupervised data table 142 and the supervised data table 143 in the storage unit 140.

[0060] The receiving unit 152 receives labels of unit actions included in element works from the input unit 120 operated by a manager or the like, and sets the received labels to the corresponding unit actions. The received labels are labels that identify element works or waste labels.

[0061] For example, the receiving unit 152 displays a display screen of each unit action included in the supervised data table 143 on the display unit 130. An administrator referring to the display unit 130 operates the input unit 120 to select a unit action and input a label for the selected unit action. The receiving unit 152 sets and updates the label corresponding to the unit action in the supervised data table 143 based on the unit action and the input label.

[0062] Extraction unit 153 extracts feature vectors based on the work video data. For example, in the "learning phase," extraction unit 153 extracts feature vectors based on the work video data stored in unsupervised data table 142, and outputs the extracted feature vectors to observation probability learning unit 154. In the "estimation phase," extraction unit 153 extracts feature vectors based on the work video data stored in video buffer 145, and outputs the extracted feature vectors to estimation unit 156.

[0063] Here, an example of the process by which the extraction unit 153 extracts feature vectors from work video data will be described. A person detection technique is applied to each frame constituting the work video data to detect a person's area (e.g., a bounding box), and the detected areas of the same person are tracked by associating them between frames. When multiple person areas are detected in a single frame, the extraction unit 153 identifies the area of ​​the person to be determined based on the size of the area, the position of the area within the frame, and the like. The extraction unit 153 performs image processing on the image within the person area detected in each frame, and calculates posture information based on the joint positions of the person and their connection relationships, and the like. The extraction unit 153 creates time-series posture information by associating the posture information calculated for each frame with the time information associated with the frame and arranging them.

[0064] The extraction unit 153 calculates time-series motion information for each body part from the time-series posture information. The motion information may be, for example, the degree of bending and the bending speed of each body part. Each body part may be, for example, an elbow, a knee, etc. The extraction unit 153 also extracts a feature vector whose elements are values ​​obtained by averaging the motion information within the window in the time direction at regular time intervals using a sliding time window. For example, the extraction unit 153 may set values ​​for each dimension of the feature vector so that it corresponds to the averaged bending speed, degree of bending, etc. of a specific body part.

[0065] The observation probability learning unit 154 trains the first model 80 based on the feature vectors extracted by the extraction unit 153 and based on the working video data in the unsupervised data table 142. For example, the first model 80 is a Gaussian mixture model (hereinafter referred to as a GMM (Gaussian Mixture Model)) model or the like.

[0066] The observation probability learning unit 154 calculates the observation probability of each unit action using a Gaussian mixture distribution model. Specifically, the observation probability learning unit 154 estimates parameters of a GMM in which as many Gaussian distributions as there are actions are mixed, by clustering the feature vectors passed from the extraction unit 153. Then, the observation probability learning unit 154 assigns each Gaussian distribution constituting the GMM whose parameters have been estimated as a probability distribution representing the observation probability of each action.

[0067] FIG. 8 is a diagram illustrating the processing of the observation probability learning unit. In the example shown in FIG. 8, the feature space V is represented by axes of a first feature, a second feature, and an n-th feature. The first feature, the second feature, and the n-th feature correspond to the respective dimensions of the feature vector. For example, the position of each feature vector in the feature space V is indicated by a square mark in FIG. 8.

[0068] When the observation probability learning unit 154 clusters the feature vectors, they are classified into clusters 45-1, 45-2, 45-3, and 45-6. For example, the feature vectors classified into cluster 45-1 are classified into clusters 45-2, 45-3, and 45-6. １ Each feature vector classified into the cluster 45-2 corresponds to the unit action "m ２ Each feature vector classified into the cluster 45-3 corresponds to the unit action "m ３ Each feature vector classified into the cluster 45-6 corresponds to the unit action "m ６ " is the feature vector corresponding to ".

[0069] The clustering result shown in FIG. 8 corresponds to the result of training the first model 80. When a feature vector to be estimated is input to the trained first model 80, the first model 80 outputs the observation probability of each unit action based on the feature vector to be estimated and the distance between each cluster. For example, if the distance between the feature vector to be estimated and cluster 45-1 is shorter than the distance between the feature vector to be estimated and the other clusters, the unit action "m １ " is a probability that the observation probability of " is greater than the observation probability of other unit actions.

[0070] Returning to the explanation of Fig. 7, the transition probability learning unit 155 trains the transition probabilities of the second model corresponding to each element task based on the supervised data table 143. The transition probability learning unit 155 trains the transition probabilities of edges between state nodes of unit actions using maximum likelihood estimation, an Expectation-Maximization algorithm (EM algorithm), or the like.

[0071] An example of the processing of the transition probability learning unit 155 will be described with reference to FIG. 9. FIG. 9 is a diagram for explaining an example of the processing of the transition probability learning unit. For example, in the supervised data table 143, if the unit action m ２ , m ５ , m ６ , m ８ In this case, the transition probability learning unit 155 calculates the unit action m ２ , m ５ , m ６ , m ８ The transition probabilities of the edges between the state nodes of the second model 90A are trained by the sequence of the above.

[0072] In addition, in the supervised data table 143, the unit action m of the time series included in the section T10-2 ６ , m ９ , m ５ is labeled as element work B. In this case, the transition probability learning unit 155 ６ , m ９ , m ５The transition probabilities of the edges between the state nodes of the second model 90B are trained by the sequence of the above.

[0073] The transition probability learning unit 155 trains the transition probability of the second model corresponding to each element task by repeatedly executing the above process based on the relationship between the label and the sequence of unit tasks registered in the supervised data table 143. The order of the element tasks is specified in advance, and the second models are connected based on the specified information. If the order of the element tasks is element tasks A, B, C, and D, the second models are connected in the following order: the second model corresponding to element task A, the second model corresponding to element task B, the second model corresponding to element task C, and the second model corresponding to element task D.

[0074] Furthermore, the transition probability learning unit 155 identifies the duration of each element work based on a section of unit actions in which the same element work label is set consecutively among the multiple unit actions registered in the supervised data table 143. The transition probability learning unit 155 identifies a probability distribution of the duration based on the identified duration of each element work, and sets the identified probability distribution in the second model of each element work.

[0075] Here, when the label of a unit operation set in the supervised data table 143 is changed by the administrator operating the input unit 120, the transition probability learning unit 155 updates the transition probability of the second model of the corresponding element work. The supervised data table 143 stores information on the label change history.

[0076] The process of updating the transition probabilities of the second model by the transition probability learning unit 155 will be described with reference to FIG. 5. For example, when the sequence of unit actions “m ２ , m ５ , m ６ , m ８ It is assumed that the label "Element Task A" has been set to ". The manager can change the sequence of unit operations "m ２ , m ５ , m ６ , m ８ " Among them, the unit action "m ５ "," "m ６If it is found that the unit action "m" is a useless unit action, the input unit 120 is operated to remove the unit action "m" from the list. ５ "," "m ６ For example, the transition probability learning unit 155 scans the unit actions in a time series to identify the labels to which the waste label has been set.

[0077] The transition probability learning unit 155 calculates the sequence of unit actions "m ２ , m ５ , m ６ , m ８ " Among them, the unit action "m ５ "," "m ６ The transition probability learning unit 155 executes the following process when the label of the unit action m ２ From the state node of ５ The transition probability of the edge to the state node of unit action m ５ From the state node of ６ The transition probability learning unit 155 also reduces the transition probability of the edge to the state node m ６ From the state node of ８ Decrease the transition probability of the edge to the state node of

[0078] The transition probability learning unit 155 determines whether the information processing device 100 has a sequence of unit actions "m ２ , m ５ , m ６ , m ８ ", then the unit action "m ５ , m ６ A sequence of unit actions, excluding "m ２ , m ８ In this case, the transition probability learning unit 155 performs re-learning using maximum likelihood estimation or the EM algorithm.

[0079] The transition probability learning unit 155 may construct the machine learning model shown in Fig. 10 based on the observation probability of each unit action calculated by the observation probability learning unit 154, the transition probability of edges between unit actions (state nodes), and the probability distribution of the duration set for each element task. The machine learning model corresponds to the above-mentioned first model and second model.

[0080] FIG. 10 is a diagram illustrating an example of a machine learning model. The machine learning model 41 illustrated in FIG. 10 is a hidden semi-Markov model (HSMM) in which a second model corresponding to each elemental behavior transitions in the order of the elemental behaviors after a set duration. In FIG. 10, O1, O2, ..., O8 represent observation probabilities calculated by the observation probability learning unit 154. Furthermore, the transition probabilities associated with the arrows between the actions m1, m2, and m3 included in each of the elemental behaviors a1, a2, and a3 correspond to the transition probabilities calculated by the transition probability learning unit 155. Furthermore, d1, d2, and d3 represent the duration of each elemental behavior.

[0081] Returning to the description of Fig. 7, in the estimation phase, the estimation unit 156 estimates the elemental work of the worker in each section. The estimation unit 156 acquires time-series feature vectors from the extraction unit 153. These feature vectors are feature vectors extracted from the work video data in the video buffer 145.

[0082] The estimation unit 156 estimates the sequence of unit actions in a time series by inputting the time series feature vectors into the first model 80. The estimation unit 156 compares the sequence of the estimated unit matrices with each of the second models 90A, 90B (second models of other elemental actions), transitions the state nodes, and sequentially identifies elemental actions corresponding to the sequence of unit actions based on the transition probability of the edge that actually transitioned and various constraints. The various constraints include constraints on task time, constraints on task order, etc. The estimation unit 156 displays the estimation results of the elemental actions on the display unit 130.

[0083] The estimation unit 156 registers the estimation result of the element matrix and the sequence of unit actions included in the element matrix in the estimation result table 144. The estimation unit 156 assigns a label identifying an element work to each unit action. The transition probability learning unit 155 may re-learn the second model based on the information registered in the estimation result table 144. The administrator may operate the input unit 120 to refer to the information in the estimation result table 144 and change the label set for each unit action in the estimation result table 144.

[0084] The generation unit 157 performs the following processing to generate information to be registered in the supervised data table 143. Here, a case will be described in which the generation unit 157 cooperates with the reception unit 152 and extraction unit 153 to generate information to be registered in the supervised data table 143 from the work video data registered in the unsupervised data table 142.

[0085] The extraction unit 153 extracts feature vectors based on the work video data stored in the unsupervised data table 142, and outputs the extracted feature vectors to the generation unit 157. The generation unit 157 estimates the sequence of unit actions in time series by inputting the time-series feature vectors into the first model 80. The reception unit 152 receives labels of element tasks for each time-series interval for the work video data stored in the unsupervised data table 142 from the administrator operating the input unit 120, and outputs information on the received intervals and labels to the generation unit 157.

[0086] The generation unit 157 assigns an element work label to each unit action based on the sequence of unit actions in time series and the labels of element works for each section. The generation unit 157 registers information on the time series unit actions for which the labels have been assigned in the supervised data table 143.

[0087] Here, the case where the generation unit 157 generates information on labeled unit actions based on the work video data registered in the unsupervised data table 142 has been described, but the present invention is not limited to this. The generation unit 157 may generate information on labeled unit actions based on the work video data transmitted from the camera 15 and register the information in the supervised data table 143.

[0088] Next, an example of the processing procedure of the information processing device according to this embodiment will be described. Figures 11 to 13 are flowcharts showing the processing procedure of the information processing device according to this embodiment. First, Figure 11 will be described. As shown in Figure 11, the acquisition unit 151 of the information processing device 100 acquires work video data (step S101).

[0089] The receiving unit 152 of the information processing device 100 receives labels of elemental tasks for each time-series section of the work video data (step S102). The generating unit 157 of the information processing device 100 estimates time-series unit actions based on the work video data (step S103).

[0090] The generating unit 157 registers the labeled time-series unit actions in the supervised data table 143 (step S104).

[0091] 12 will be described. The extraction unit 153 of the information processing device 100 acquires work video data from the unsupervised data table 142 (step S201). The extraction unit 153 extracts a feature vector based on the work video data (step S202).

[0092] The observation probability learning unit 154 of the information processing device 100 trains (performs unsupervised learning) the observation probability of the first model 80 based on the feature vector extracted by the extraction unit 153 (step S203).

[0093] The transition probability learning unit 155 of the information processing device 100 acquires the sequence of unit actions and the label from the supervised data table 143 (step S204). The transition probability learning unit 155 selects a second model corresponding to the label (step S205). The transition probability learning unit 155 trains the transition probabilities of the second model (supervised learning) based on the sequence of unit actions (step S206).

[0094] If unprocessed data exists in the supervised data table 143 (step S207, Yes), the transition probability learning unit 155 proceeds to step S204. On the other hand, if unprocessed data does not exist in the supervised data table 143 (step S207, No), the transition probability learning unit 155 ends the process.

[0095] 13 will be described. The transition probability learning unit 155 of the information processing device 100 receives a label update request (step S301). The transition probability learning unit 155 acquires information about unit actions from the supervised data table 143 (step S302).

[0096] The transition probability learning unit 155 displays screen information in which each unit action is associated with a label on the display unit 130 (step S303). The transition probability learning unit 155 accepts a change to the label related to the unit action of the specified element work (step S304).

[0097] The transition probability learning unit 155 selects a second model corresponding to the specified element work (step S305). The transition probability learning unit 155 changes the transition probability of the second model based on the label type (step S306).

[0098] Next, a description will be given of the effects of the information processing device 100 according to this embodiment. When an unnecessary unit action is specified in a sequence of unit actions corresponding to element tasks, the information processing device 100 can improve the accuracy of identifying a person's element tasks by updating the transition probability of the second model of the corresponding element task.

[0099] For example, as described in Fig. 6, the detection result by the information processing device 100 is detection result 33. Comparing the correct detection result 31 with detection result 33, the order of the elemental tasks is the same, and the lengths of the sections of each elemental task are also approximately the same. In other words, the information processing device 100 according to this embodiment can improve the accuracy of identifying a person's task compared to the prior art.

[0100] The information processing device 100 calculates a feature vector based on the work video data and inputs the feature vector into the first model 80 generated by unsupervised learning, thereby identifying the unit movements. This makes it possible to appropriately classify each unit movement included in the work with variations.

[0101] When an unnecessary unit action is specified, the information processing device 100 updates the transition probabilities between state nodes of the second model of the corresponding elemental task, thereby improving the accuracy of identifying elemental tasks of a person.

[0102] The information processing device 100 identifies, among the plurality of unit actions, unit actions to which a waste label has been assigned, and updates the transition probabilities corresponding to the unit actions to which the waste label has been assigned among the plurality of edges connecting the plurality of unit actions included in the second model, thereby enabling the transition probabilities of the second model to be updated efficiently.

[0103] The information processing device 100 trains the transition probabilities of edges connecting the plurality of unit actions included in the second model based on the plurality of unit actions in a time series, excluding the unit actions to which waste labels are assigned, from among the plurality of unit actions. This allows the second model to be retrained using appropriate supervised data, thereby improving the accuracy of the second model.

[0104] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 14 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0105] 14, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 15, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0106] The hard disk drive 207 stores an acquisition program 207 a, a reception program 207 b, an extraction program 207 c, an observation probability learning program 207 d, a transition probability learning program 207 e, an estimation program 207 f, and a generation program 207 g. The CPU 201 reads out each of the programs 207 a to 207 g and loads them into the RAM 206.

[0107] The acquisition program 207a functions as an acquisition process 206a. The reception program 207b functions as a reception process 206b. The extraction program 207c functions as an extraction process 206c. The observation probability learning program 207d functions as an observation probability learning process 206d. The transition probability learning program 207e functions as a transition probability learning process 206e. The estimation program 207f functions as an estimation process 206f. The generation program 207g functions as a generation process 206g.

[0108] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the reception process 206b corresponds to the processing of the reception unit 152. The processing of the extraction process 206c corresponds to the processing of the extraction unit 153. The processing of the observation probability learning process 206d corresponds to the processing of the observation probability learning unit 154. The processing of the transition probability learning process 206e corresponds to the processing of the transition probability learning unit 155. The processing of the estimation process 206f corresponds to the processing of the estimation unit 156. The processing of the generation process 206g corresponds to the processing of the generation unit 157.

[0109] It should be noted that each of the programs 207a to 207g does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207g.

[0110] REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Model table 142 Unsupervised data table 143 Supervised data table 144 Estimation result table 145 Video buffer 150 Control unit 151 Acquisition unit 152 Reception unit 153 Extraction unit 154 Observation probability learning unit 155 Transition probability learning unit 156 Estimation unit 157 Generation unit

Claims

1. Obtain footage of a person working, receiving a label indicating an elemental task of a person for each section of the time series of the acquired video; A learning process is performed on a computer to learn transition probabilities of features per unit time constituting an elemental work based on the received labels; The learning process changes a transition probability of a feature quantity in a section corresponding to a label based on the type of the label when a specific type of label is assigned to the whole or part of the element work. A machine learning program characterized by

2. The machine learning program according to claim 1, further comprising a process of calculating a feature vector based on the video and identifying a unit action per unit time based on the calculated feature vector, wherein the learning process learns a transition probability of an edge connecting a plurality of unit actions included in a model corresponding to the element work.

3. 3. The machine learning program according to claim 2, characterized in that the learning process identifies a unit action to which the specific type label has been assigned among a plurality of unit actions in a time series corresponding to the element work, and updates a transition probability corresponding to the unit action to which the specific type label has been assigned among a plurality of edges connecting a plurality of unit actions included in the model.

4. 3. The machine learning program according to claim 2, characterized in that the learning process learns transition probabilities of edges connecting a plurality of unit actions included in the model, based on a plurality of unit actions in a time series corresponding to the element work, excluding unit actions to which the specific type of label has been assigned.

5. Obtain footage of a person working, receiving a label indicating an elemental task of a person for each section of the time series of the acquired video; A computer executes a learning process to learn transition probabilities of features per unit time constituting an elemental task based on the received labels; The learning process changes a transition probability of a feature quantity in a section corresponding to a label based on the type of the label when a specific type of label is assigned to the whole or part of the element work. A machine learning method comprising:

6. Obtain footage of a person working, receiving a label indicating an elemental task of a person for each section of the time series of the acquired video; a control unit that executes a learning process to learn transition probabilities of feature quantities per unit time constituting an elemental work based on the received labels; The learning process changes a transition probability of a feature in a section corresponding to a label based on the type of the label when a specific type of label is assigned to the whole or part of the element work.

23. An information processing apparatus comprising: