Activity interval estimation model construction device, activity interval estimation model construction method, and activity interval estimation model construction program

Abstract

In a hidden semi-Markov model, observation probabilities for each type of behavior in a plurality of first hidden Markov models are trained through unsupervised learning. The hidden semi-Markov model includes a plurality of second hidden Markov models including a plurality of first hidden Markov models that each have types of human behavior as states, and each of the plurality of second hidden Markov models have, as states, actions defined by combining a plurality of behaviors. Second supervised data is imparted by fixing the trained observation probabilities and diluting inputted first supervised data, and transition probabilities for first hidden Markov model behaviors are trained through supervised learning in which the second supervised data is used. A hidden semi-Markov model, which is a model that estimates action intervals by using the trained observation probabilities and transition probabilities, is built. The first supervised data is diluted by adding training information to data generated by implementing at least one of oversampling in the time direction and oversampling in the feature space.

Description

[Technical field]

[0001] The present disclosure relates to an activity interval estimation model construction device, an activity interval estimation model construction method, and an activity interval estimation model construction program. [Background technology]

[0002] Advances in deep learning technology have made it possible to accurately recognize postures from images of people captured with normal RGB cameras, and various research and development efforts are being conducted to estimate human behavior using this recognition information. Under these circumstances, efforts are being made to estimate the time period in which a specified behavior occurred from time-series data of postures detected from human images. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ryuichi Yamamoto, Shinji Sako, Tadashi Kitamura, "Real-time Alignment of Musical Audio Signals and Sheet Music by Combining Hidden Semi-Markov Models and Linear Dynamical Systems," Research Report on Music Information Science (MUS), 2012 [Non-Patent Document 2] Shun-ZhengYu, "Hidden semi-Markov models", Artificial Intelligence, Volume 174, Issue 2, February 2010, pp. 215-243 [Non-Patent Document 3] Kei Wakabayashi and Takao Miura, "Fast Parameter Estimation for Hierarchical Hidden Markov Models," IEICE Transactions on Information and Communication Engineers, 2011 [Non-Patent Document 4] "Fujitsu develops AI technology "Actlyzer" that recognizes various human behaviors from video," [online], November 25, 2019, Fujitsu Ltd., [Retrieved January 19, 2020], Internet (URL: https: / / pr.fujitsu.com / jp / news / 2019 / 11 / 25.html) Summary of the Invention [Problem to be solved by the invention]

[0004] The cost of creating supervised data training information when training a model to estimate the time interval of an action is high.

[0005] An object of the present disclosure, as one aspect, is to efficiently construct an action interval estimation model. [Means for solving the problem]

[0006] In one embodiment, in a hidden semi-Markov model, the observation probability for each type of motion of a plurality of first hidden Markov models is learned by unsupervised learning. The hidden semi-Markov model includes a plurality of second hidden Markov models including a plurality of first hidden Markov models each having a state corresponding to a type of human motion, and each of the plurality of second hidden Markov models has a state corresponding to an action determined by combining a plurality of motions. The learned observation probability is fixed, and the input first supervised data is augmented to obtain second supervised data, and the transition probability of the motion of the first hidden Markov model is learned by supervised learning using the second supervised data. The learned observation probability and transition probability are used to construct a hidden semi-Markov model that is a model for estimating a section of a motion. The first supervised data is augmented by adding supervised information of the first supervised data to each of the data generated by performing at least one of oversampling in the time direction and oversampling in the feature space. Effect of the Invention

[0007] One aspect of the present disclosure is that it is possible to efficiently construct an action interval estimation model. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a conceptual diagram illustrating a hidden semi-Markov model according to the present embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a functional configuration of the present embodiment. [Diagram 3]3 is a conceptual diagram illustrating a state of a first hidden Markov model according to the present embodiment. FIG. [Figure 4] FIG. 1 is a conceptual diagram illustrating padding of supervised data. [Diagram 5] FIG. 1 is a conceptual diagram illustrating padding of supervised data. [Figure 6] FIG. 1 is a conceptual diagram illustrating padding of supervised data. [Figure 7] FIG. 1 is a conceptual diagram illustrating padding of supervised data. [Figure 8] FIG. 1 is a conceptual diagram illustrating padding of supervised data. [Figure 9] FIG. 1 is a conceptual diagram illustrating padding of supervised data. [Figure 10] FIG. 2 is a block diagram illustrating a hardware configuration of the present embodiment. [Figure 11] 11 is a flowchart illustrating the flow of a movement interval estimation model construction process; [Figure 12] 11 is a flowchart illustrating the flow of a feature vector extraction process. [Figure 13] 13 is a flowchart illustrating the flow of a supervised data augmentation process. [Figure 14] 13 is a flowchart illustrating the flow of a behavior interval estimation process. [Figure 15] FIG. 1 is a conceptual diagram illustrating the behavior of the related art. [Figure 16] FIG. 1 is a conceptual diagram illustrating a hierarchical hidden Markov model of the related art. [Figure 17] FIG. 1 is a conceptual diagram illustrating an overview of the related art. [Figure 18] FIG. 1 is a conceptual diagram illustrating an overview of the present embodiment. [Figure 19] 1 is a conceptual diagram illustrating fluctuations in observation data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In this embodiment, as an example of an action interval estimation model for estimating a time interval in which a human action occurs, a hidden semi-Markov model (hereinafter referred to as HSMM (Hidden semi-Markov model)) as shown in Fig. 1 is constructed. In addition to parameters of a hidden Markov model (hereinafter referred to as HMM (Hidden Markov model)), the HSMM has a probability distribution of duration of each state as a parameter.

[0010] The HSMM of this embodiment includes a plurality of first HMMs in which each of the human movements is a state, and a second HMM in which each of the second HMMs is an action determined by combining a plurality of movements. m1, m2, and m3 are examples of movements, and a1, a2, and a3 are examples of actions. An action is a combination of a plurality of movements, and an action is a combination of a plurality of postures.

[0011] When the HSMM constructed by setting parameters is given time-series sensor data generated by detecting a person's posture, the HSMM estimates the optimal time interval of an action (hereinafter, called an action interval). d1, d2, and d3 are examples of action intervals.

[0012] The parameters of an HMM include observation probabilities and transition probabilities. O1, ..., O8 are examples of observation probabilities, and transition probabilities are probabilities corresponding to arrows connecting states. Observation probability is the probability that a certain feature is observed in each state, and transition probability is the probability of transitioning from one state to another. If the order of transitions is fixed, transition probabilities are not necessary. Note that the number of actions and behaviors, i.e., the number of first HMMs and second HMMs, are examples and are not limited to the numbers exemplified in FIG. 1.

[0013] 2 is an example of a functional block diagram of the activity interval estimation model construction device 10 of this embodiment. The activity interval estimation model construction device 10 includes an observation probability learning unit 11, a transition probability learning unit 12, and a construction unit 13. The observation probability learning unit 11 learns the observation probability of HSMM, which is an example of an activity interval estimation model, using unsupervised data, as described below.

[0014] In this embodiment, the subject is a limited action for achieving a certain task goal. Such an action is, for example, an action in a routine task performed on a factory line, and has the following characteristics. Property 1: The differences between the actions that make up a task are differences in combinations of a limited number of actions. Property 2: Multiple postures observed when performing the same task are similar.

[0015] In this embodiment, all actions are composed of actions included in one action group based on property 1. As illustrated in Fig. 3, the action group includes, for example, three actions m11, m12, and m13.

[0016] For example, the action m11 may be "raise the arm", the action m12 may be "lower the arm", and the action m13 may be "stretch the arm forward". The number of actions included in the action group is not limited to the example in Fig. 3. Furthermore, the number of actions included in each action is not limited to the example in Fig. 3.

[0017] In the HMM in Figure 3, the observation probability of each action corresponding to the dashed arrow does not depend on the action, so it can be learned from unsupervised data of the action interval. Learning is performed using, for example, machine learning, neural networks, deep learning, etc.

[0018] In detail, the model used in the unsupervised learning of the observation probability may be a Gaussian mixture distribution (hereinafter referred to as GMM (Gaussian Mixture Model)). It is assumed that one of the actions is probabilistically selected for each observation, and that each observation is generated by a Gaussian distribution for that action. This is an assumption that differs from supervised learning, which does not use the time-series dependency of the observations. The parameters of each Gaussian distribution of the learned GMM are assigned to a Gaussian distribution that is the probability distribution of the observation probability for each action.

[0019] The transition probability learning unit 12 learns the transition probability of the actions of the first HMM using learning data having teacher information (hereinafter referred to as supervised data), as described below. The teacher information is information that gives the correct answer of the time interval during which each action occurs for the posture time series data. The learning is performed using, for example, maximum likelihood estimation or an Expectation-Maximization algorithm (EM algorithm) (other methods such as machine learning, neural network, and deep learning may also be used).

[0020] Generating supervised data takes time and effort. Therefore, in this embodiment, the observation probability learned by the observation probability learning unit 11 is fixed, and the transition probability is learned from existing supervised data.

[0021] 4, data of existing supervised data, which is an example of first supervised data, is used as seed data SD, and the data is augmented by oversampling. In this embodiment, for example, oversampling is performed in the time direction, and then oversampling is performed in the feature space.

[0022] Oversampling in the time direction will now be described. In oversampling in the time direction, for example, time expansion / contraction related to the length of time it takes for a motion that differs from person to person is taken into consideration. The details are as follows. (1) As shown in Fig. 5, for each time point in the observed sequence of human movements, a random number is generated that represents the strength of growth of the feature at that time point. The vertical lines at each time point in Fig. 5 represent the strength of growth generated by the random numbers corresponding to the original parameters. (2) The strength of the stretch at each time is attenuated and propagated to the times before and after the time. The strength of the stretch is attenuated so that it becomes 0 at a time a specified number of times away. In the example of Figure 5, as shown by the dashed line, it is attenuated so that it becomes 0 at a time three times away. The attenuation does not have to be linear. (3) Of the original stretch strength at each time and the propagated stretch strength corresponding to the parameters propagated from the previous and next times, the feature value at the time corresponding to the greatest strength is selected as the feature value at that time. In the example of Figure 5, at time 1, the original stretch strength is greatest, so the feature value at time 1, which is the original feature value, is selected, and at time 2, the stretch strength propagated from time 1 is greatest, so the feature value at time 1 is selected. At time 3, the stretch strength propagated from time 1 is greatest, so the feature value at time 1 is selected, and at time 4, the original stretch strength is greatest, so the feature value at time 4, which is the original feature value, is selected.

[0023] We will now explain oversampling in the feature space. According to property 2 above, postures for the same task are similar, so by adding noise, it is possible to generate data with variance similar to the variance for each actual observation, as shown in the example of Figure 6.

[0024] The supervised data is augmented by commonly applying the supervised information TI of the seed data SD to each of the augmented data. The augmented supervised data, which is an example of the second supervised data, is used to learn the transition probabilities of the multiple actions of the first HMM through supervised learning.

[0025] In oversampling, noise is generated and added to the feature values ​​at each time. For example, noise generated from a multivariate Gaussian distribution with a covariance that is a constant multiple of the covariance of the sample group of the identified movement may be added. Alternatively, the center distance d from the sample group of the identified movement to the sample group of the movement with the closest center distance may be calculated, and noise generated from an isotropic Gaussian distribution (with a diagonal covariance matrix) in which the standard deviation in each axis direction of the feature space is a constant multiple of d may be added.

[0026] In this embodiment, noise related to the speed of each body part of the person performing the movement is added to the feature value of the movement for each body part. For example, the diagonal components, which are the variance components of the covariance matrix of the Gaussian distribution, are changed for each body part of the person performing the movement. In detail, the standard deviation σ of the feature value, which is the posture component of the feature vector of the body part i (i is a natural number), is i '(variance σ i ' 2 ) is the angular velocity ω i , the base standard deviation value σ i (variance σ i 2 ), and is calculated using formula (1) with a constant coefficient k. σ i '=σ i +kω i …(1)

[0027] σ i and k are constants experimentally determined in advance and do not change for each body part. As shown in the second term of equation (1), noise, i.e., posture variation, is increased in proportion to the magnitude of the angular velocity. For example, the horizontal axis of FIG. 7 represents feature value 1, which is the posture component of body part 1, and the vertical axis represents feature value 2, which is the posture component of body part 2.

[0028] In Fig. 7, the feature space is represented in two dimensions, but the number of dimensions may be more than two. In Fig. 7, the ellipses represent the contours of the probability distribution (Gaussian distribution) that the samples represented by the points in the feature space of the actions m21, m22, and m23 are observed. The closer to the center of the ellipse, the higher the probability.

[0029] When the angular velocity components of the movements of body part 1 and body part 2 are approximately similar, noise of approximately the same magnitude is added to both the vertical and horizontal directions, as shown on the left in Fig. 7. On the other hand, when the angular velocity component of the movements of body part 1 is larger than the angular velocity component of the movements of body part 2, noise that is larger in the horizontal direction than in the vertical direction is added, as shown on the right in Fig. 7.

[0030] Oversampling in the time direction makes it possible to respond to changes in the time direction. That is, even when performing the same task, a certain movement (movement feature) may be observed to be short or long depending on whether the person moves quickly or slowly. In the case of fast movements, a certain movement may not be observed.

[0031] For example, as shown in the left side of Fig. 8, worker A spends about three hours on movement 2, while as shown in the upper right side of Fig. 8, worker B spends about four hours on movement 2, and as shown in the lower right side of Fig. 8, worker C spends about one hour on movement 2. By performing oversampling in the time direction, it is possible to pad out samples that are stretched or shrunk in time like this.

[0032] Oversampling in the feature space can deal with variations in feature values ​​representing posture. For example, as shown in the left side of Fig. 9, when the moving speed of the first arm is high and the moving speed of the second arm is low, the change in posture of the first arm is also large in proportion to the speed, and therefore the variation in feature values ​​is also large, as shown in the right side of Fig. 9.

[0033] On the other hand, the change in posture of the second arm is small in proportion to the speed, and therefore the variance in the feature values ​​is also small. By oversampling in the feature space, it is possible to augment samples with different variances in the feature values ​​for different body parts.

[0034] Both the time-direction oversampling and the feature-direction oversampling may be performed, or only one of them may be performed. When only the feature-direction oversampling is performed, noise related to the speed of each body part of the person performing the action is added to the feature value of each body part at each time.

[0035] The construction unit 13 constructs an HSMM as illustrated in FIG. 1 using the observation probability learned by the observation probability learning unit 11 and the state transition probability learned by the transition probability learning unit 12. O1, O2, ..., O8 represent the observation probabilities learned by the observation probability learning unit 11, and the arrows between the actions m1, m2, and m3 included in each of the actions a1, a2, and a3 correspond to the state transition probabilities learned by the transition probability learning unit 12. d1, d2, and d3 represent the duration of each action, and the probability distribution of the duration is determined from the duration of the action in the teacher information. For example, the probability distribution of the duration may be a uniform distribution within a certain range. The sensor data generated by detecting the posture of a person with a sensor is applied to the constructed HSMM to estimate the action interval, which is the time interval of each action. Details of the estimation will be described later.

[0036] The movement interval estimation model construction device 10 of this embodiment has the following features. 1. The observation probability of actions common to all actions in the first HMM is learned through unsupervised learning. 2. The transition probabilities between the actions of the first HMM are learned through supervised learning using augmented supervised data from the supervised seed data.

[0037] As an example, as shown in FIG. 10, the action interval estimation model construction device 10 includes a CPU (Central Processing Unit) 51, a primary storage device 52, a secondary storage device 53, and an external interface 54. The CPU 51 is an example of a processor which is hardware. The CPU 51, the primary storage device 52, the secondary storage device 53, and the external interface 54 are connected to each other via a bus 59. The CPU 51 may be a single processor or may be multiple processors. In addition, instead of the CPU 51, for example, a GPU (Graphics Processing Unit) may be used.

[0038] The primary storage device 52 is, for example, a volatile memory such as a random access memory (RAM). The secondary storage device 53 is, for example, a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD).

[0039] The secondary storage device 53 includes a program storage area 53A and a data storage area 53B. The program storage area 53A stores, for example, a program for constructing a movement interval estimation model. The data storage area 53B stores, for example, supervised data, unsupervised data, learned observation probabilities, transition probabilities, and the like.

[0040] The CPU 51 reads out the movement interval estimation model construction program from the program storage area 53A and deploys it in the primary storage device 52. The CPU 51 loads and executes the movement interval estimation model construction program to operate as the observation probability learning unit 11, the transition probability learning unit 12, and the construction unit 13 in FIG.

[0041] Note that programs such as the activity interval estimation model construction program may be stored in an external server and loaded into the primary storage device 52 via a network. Programs such as the activity interval estimation model generation program may be stored in a non-transitory recording medium such as a DVD (Digital Versatile Disc) and loaded into the primary storage device 52 via a recording medium reading device.

[0042] An external device is connected to the external interface 54, and the external interface 54 controls transmission and reception of various information between the external device and the CPU 51. Fig. 10 shows an example in which a display 55A and an external storage device 55B are connected to the external interface 54. The external storage device 55B stores, for example, supervised data, unsupervised data, and the constructed HSMM. The display 55A visually displays, for example, the constructed HSMM model.

[0043] The movement interval estimation model construction device 10 may be, for example, a personal computer, a server, or a computer on the cloud.

[0044] 11 illustrates an example of the flow of the action interval estimation model construction process. In step 101, the CPU 51 extracts feature vectors representing movements that are chains of human postures from the learning data, as described below. In step 102, the CPU 51 classifies the feature vectors extracted in step 101 into elemental movements by clustering (GMM parameter estimation), and learns the observation probability of each movement by unsupervised learning.

[0045] In step 103, the CPU 51 pads the supervised data by adding supervision information of the supervised seed data to data generated by oversampling the supervised seed data, as described below. In step 104, the CPU 51 assigns feature vectors to the supervised data for each time interval of each action given by the supervision information.

[0046] In step 105, the CPU 51 uses the series of feature vectors in the time interval allocated in step 104 as observed data and the supervised data augmented in step 103 to learn the transition probabilities of the actions of the first HMM through supervised learning.

[0047] In step 106, the CPU 51 sets a uniform distribution within a predetermined range for the duration of each action given in the teacher information as a probability distribution of the duration of each action. The CPU 51 constructs an HSMM using the observation probability learned in step 102 and the transition probability learned in step 105. In the setting of step 106, an HSMM is constructed in which the actions of the second HMM transition in the order of each action given in the teacher information after a certain period of time has elapsed. The constructed HSMM may be stored, for example, in the data storage area 53B.

[0048] Fig. 12 illustrates the details of the feature vector extraction process in step 101 in Fig. 11. In step 151, the CPU 51 detects and tracks people from the data used for learning, thereby acquiring posture information of the people. In step 152, when the posture information acquired in step 151 includes posture information of multiple people, the CPU 51 acquires time series data of posture information to be analyzed from the time series data of posture information. The posture information to be analyzed is selected based on the size of a bounding box surrounding a person, time, etc.

[0049] In step 153, CPU 51 acquires time series data of motion information for each part of the body from the time series data of posture information acquired in step 152. The time series of motion information may be, for example, the degree of bending of each part, the bending speed, etc. Each part may be, for example, an elbow, a knee, etc.

[0050] In step 154, the CPU 51 calculates a feature vector by averaging the motion information in step 153 within the window in the time direction at regular time intervals using a sliding time window.

[0051] Fig. 13 illustrates the flow of the supervised data augmentation process in step 103 in Fig. 11. In step 251, the CPU 51 generates a random number representing the strength of growth of the feature at each time point in the observation data (observation time series of human movements). In step 252, the CPU 51 propagates the value of the growth strength generated at each time point to the times before and after the time point while attenuating the value.

[0052] In step 253, the CPU 51 selects the feature value of the observation data at the time corresponding to the maximum stretch strength among the stretch strength at the time and the stretch strength values ​​propagated from the other times as the feature value at the time. In step 254, the CPU 51 calculates the covariance matrix of the Gaussian distribution based on the angular velocity values ​​of each body part.

[0053] In step 255, the CPU 51 adds noise generated by the Gaussian distribution of the covariance matrix calculated in step 254 to each of the feature values ​​selected in step 253. The supervised data is augmented by repeating the augmentation of the supervised data.

[0054] It is also possible to repeatedly perform only the processes in steps 254 and 255. In this case, noise is added to the original feature value at each time. It is also possible to repeatedly perform only the processes in steps 251 to 253.

[0055] Fig. 14 illustrates a flow of a movement interval estimation process using the HSMM constructed in this embodiment. The movement interval estimation model construction device 10 in Fig. 10 may function as a movement interval estimation device by storing the constructed HSMM in the data storage area 53B.

[0056] In step 201, the CPU 51 extracts a feature vector from sensor data generated by detecting a person's posture with a sensor. The sensor is a device that detects a person's posture, and may be, for example, a camera, an infrared sensor, a motion capture device, etc. Step 201 in Fig. 14 is similar to step 101 in Fig. 11, and therefore a detailed description thereof will be omitted.

[0057] In step 202, the CPU 51 estimates the duration of each behavior state by comparing the series of feature vectors extracted in step 201 as observed data with the HSMM constructed in the behavior interval estimation model construction process. In step 203, the CPU 51 estimates the time interval of each behavior from the duration of each behavior state estimated in step 202.

[0058] For example, a technology that uses a video as input and recognizes specific actions in the video performs basic action recognition, elemental action recognition, and higher-level action recognition. A specific action in a video is a combination of elemental actions, which are more complex higher-level actions, while basic action recognition is posture recognition for each frame, and elemental action recognition is performing temporal and spatial recognition to recognize simple actions over a certain length of time. Higher-level action recognition is the recognition of complex actions over a certain length of time. In this technology, the action interval estimation model construction process of this embodiment and the constructed action interval estimation model can be applied to estimate action intervals.

[0059] In the related art, the HSMM may be used, in which the actions included in the behavior are not particularly limited. In the related art, it is assumed that the following actions exist, for example, as illustrated in FIG. (1) Raise your arms, (2) lower your arms, (3) extend your arms forward, (4) bring your hands close together in front of your body, (5) move forward, (6) move to the side, (7) squat, (8) stand up

[0060] Examples of actions are, for example: Action a31: (1) Raise arms → (3) Stretch arms forward → (1) Raise arms → (4) Bring hands close together in front of body → (7) Squat down, Action a32: (7) Crouch down → (4) bring both hands close together in front of the body → (8) stand up → (5) move forward → (3) stretch arms out in front, etc.

[0061] As described above, when an HMM includes a general action, i.e., a plurality of actions for which the action to be estimated is not limited, it is difficult to express the observation probability of the action with one simple probability distribution. To address this problem, there is a technique that uses a hierarchical hidden Markov model. In a hierarchical hidden Markov model, as illustrated in FIG. 16, an upper layer HMM includes a plurality of lower layer HMMs as states. Actions a51, a52, and a53 are examples of lower layer HMMs. Each of the lower layer HMMs includes an action as a state, and m51, m52, m53, m61, m62, m63, m71, and m72 are examples of actions.

[0062] In a hierarchical HMM, as illustrated in FIG. 17, learning data LD having teacher information TIL is used to learn the observation probability and transition probability of each action by supervised learning. FIG. 17 illustrates the observation probability p11 and transition probability p21 of action a51, the observation probability p12 and transition probability p22 of action a52, and the observation probability p13 and transition probability p23 of action a53. However, in a hierarchical HMM, since there are a large number of parameters and the degree of freedom of the parameters is high, a large amount of supervised data is used to learn the parameters. Creating supervised information for supervised data requires time and effort.

[0063] On the other hand, in the present disclosure, as illustrated in FIG. 18, the observation probability p1 common to each of the first HMMs corresponding to the actions of the HSMM is learned by unsupervised learning using unsupervised data LDN. The learned observation probability p1 is fixed, and the transition probabilities p21D, p22D, and p23D of each action of the first HMM are learned by supervised learning using supervised data. In the present disclosure, the supervised data is amplified and used for supervised learning by adding the supervised information TIL of the supervised data LDD to the data generated by oversampling the existing supervised data LDD. Therefore, in this embodiment, even if there is a small amount of existing supervised data, it is possible to efficiently construct an action interval estimation model.

[0064] As illustrated in the left side of Fig. 19, for example, the fluctuation of the observed data when the highly probable sequence of actions is action m31 at time t1, action m31 at time t2, action m33 at time t3, and action m32 at time t4 is illustrated. As illustrated in the top right of Fig. 19, when the movement of the actions changes and the observation at time t2 changes to be closer to action m32, the highly probable sequence of actions is action m31 at time t1, action m32 at time t2, action m33 at time t3, and action m32 at time t4.

[0065] As shown in the example at the bottom right of Fig. 19, when the speed of the movements increases, the sample at time t3 on the left of Fig. 19 is not observed, and the sequence of movements with high probability is movement m31 at time t1, movement m31 at time t2, and movement m32 at time t3. It is possible to learn in advance what kind of fluctuations may occur in response to such fluctuations and reflect this in the model as transition probabilities.

[0066] However, when there is a small amount of supervised data, it is not possible to directly learn various fluctuations, and the response to the fluctuations in the observed data is weak. In this embodiment, by performing oversampling in the time direction and oversampling in the feature space, it is possible to augment the supervised data that can respond to the fluctuations in the observed data.

[0067] In this embodiment, even if there is a small amount of existing supervised data, it is possible to model the sequence of operations assuming fluctuations in the observed data. Therefore, even if there are fluctuations in the observed data, it is possible to estimate the time interval with high accuracy.

[0068] In this embodiment, in the hidden semi-Markov model, the observation probability for each type of motion of a plurality of first hidden Markov models is learned by unsupervised learning. The hidden semi-Markov model includes a plurality of second hidden Markov models including a plurality of first hidden Markov models each having a state corresponding to a type of human motion, and each of the plurality of second hidden Markov models has a state corresponding to an action determined by combining a plurality of motions. The learned observation probability is fixed, and the input first supervised data is padded to obtain second supervised data, and the transition probability of the motion of the first hidden Markov model is learned by supervised learning using the second supervised data. The learned observation probability and transition probability are used to construct a hidden semi-Markov model, which is a model that estimates a section of a motion. The first supervised data is padded by adding supervised information of the first supervised data to each of the data generated by performing at least one of oversampling in the time direction and oversampling in the feature space.

[0069] According to the present disclosure, an action interval estimation model can be efficiently constructed. That is, for a plurality of actions that are performed in a fixed order, such as routine work in a factory, dance choreography, or martial arts form, the time interval of each action can be accurately estimated under the condition that there are constraints on the order in which the actions occur. [Explanation of symbols]

[0070] 10. Activity interval estimation model construction device 11 Observation Probability Learning Unit 12 Transition Probability Learning Unit 13 Construction Department 51 CPU 52 Primary storage 53 Secondary storage device

Claims

1. a hidden semi-Markov model including a plurality of second hidden Markov models including a plurality of first hidden Markov models each having a state corresponding to a type of human motion, and each of the plurality of second hidden Markov models having a state corresponding to an action determined by combining a plurality of the motions; and an observation probability learning unit configured to learn an observation probability for each type of motion of the plurality of first hidden Markov models by unsupervised learning; a transition probability learning unit that fixes the observation probability learned by the observation probability learning unit, pads input first supervised data to obtain second supervised data, and learns a transition probability of the operation of the first hidden Markov model by supervised learning using the second supervised data; a construction unit that constructs the hidden semi-Markov model, which is a model that estimates an interval of the action by using the observation probability learned by the observation probability learning unit and the transition probability learned by the transition probability learning unit; Including, The transition probability learning unit performs at least one of oversampling in a time direction and oversampling in a feature space on the first supervised data to generate data, and adds supervised information of the first supervised data to each of the data. A behavior interval estimation model construction device.

2. The oversampling in the time direction propagates the original parameters set randomly at each time to the previous and next times while attenuating them, At each time, a feature value of the motion corresponding to a time of a maximum parameter among the original parameters and the parameters propagated from the previous and subsequent times is selected as a feature value of the time. The action interval estimation model building device according to claim 1 .

3. The original parameters are decayed to zero at a predetermined number of times apart. The action interval estimation model constructing device according to claim 2 .

4. The oversampling on the feature space adds noise related to the speed of each body part of the person performing the action to the feature value of the action for each body part of the first supervised data. The action interval estimation model construction device according to any one of claims 1 to 3.

5. The magnitude of the noise associated with the velocity of each of the body parts is larger as each of the angular velocities of each of the body parts is larger. The action interval estimation model constructing device according to claim 4 .

6. A computer comprising: In a hidden semi-Markov model including a plurality of second hidden Markov models including a plurality of first hidden Markov models each having a state corresponding to a type of human motion, each of the plurality of second hidden Markov models having a state corresponding to an action determined by combining a plurality of the motions, an observation probability for each type of motion of the plurality of first hidden Markov models is learned by unsupervised learning; The learned observation probability is fixed, and the input first supervised data is augmented to obtain second supervised data, and a transition probability of the operation of the first hidden Markov model is learned by supervised learning using the second supervised data; constructing the hidden semi-Markov model, which is a model that estimates the behavior interval using the learned observation probability and transition probability; A method for constructing a movement interval estimation model, comprising the steps of: augmenting the first supervised data by adding supervised information of the first supervised data to each of data generated by performing at least one of oversampling in a time direction and oversampling in a feature space; A method for constructing a movement interval estimation model.

7. In a hidden semi-Markov model including a plurality of second hidden Markov models including a plurality of first hidden Markov models each having a state corresponding to a type of human motion, each of the plurality of second hidden Markov models having a state corresponding to an action determined by combining a plurality of the motions, an observation probability for each type of motion of the plurality of first hidden Markov models is learned by unsupervised learning; The learned observation probability is fixed, and the input first supervised data is augmented to obtain second supervised data, and a transition probability of the operation of the first hidden Markov model is learned by supervised learning using the second supervised data; constructing the hidden semi-Markov model, which is a model that estimates the behavior interval using the learned observation probability and transition probability; A process comprising: augmenting the first supervised data by adding supervised information of the first supervised data to each of data generated by performing at least one of oversampling in a time direction and oversampling in a feature space; A program for constructing a movement interval estimation model that causes a computer to execute processing.

Images