Action section detection program and method

Abstract

In the present invention, an action section detection device: extracts observation information from a video obtained by filming actions of a person who executes a plurality of element actions in order; and detects the end time of a detection target element action among the actions, on the basis of a second-part detection model, in a case where a second likelihood of first observation information with respect to the second-part detection model, which includes a probability model representing each element action from the start of the actions to the element action after the detection target element action, is higher than a first likelihood of the first observation information from the start time of the actions to the current time with respect to a first-part model, which includes a probability model representing each element action from the start of the actions to the detection target element action.

Description

Activity interval detection program and method

[0001] The disclosed technology relates to an action interval detection program and an action interval detection method.

[0002] Various technologies have been developed to analyze human behavior using time-series data, such as a person's body posture, extracted from video. For example, a behavior section detection device has been proposed that extracts time-series features from video of a person's behavior and estimates element behavior sections using the time-series features corresponding to candidate sections and a first machine learning model. This device calculates the relative fitness for each evaluation section using the time-series features corresponding to an evaluation section that combines element behavior sections and a second machine learning model. This device then calculates a final evaluation value from the relative fitness for each evaluation section and determines candidate sections whose final evaluation value is equal to or greater than a threshold as the behavior section to be detected.

[0003] Furthermore, for example, an action section evaluation device has been proposed that evaluates the estimation result of the segmentation of element action sections based on time series feature amounts extracted from a video of a person's action and a detection model including a probability model for each of a plurality of element actions included in the action. This device evaluates the time T corresponding to the boundary j in the segment estimation result. ｊ When the boundary j is shifted at each time based on the above, the function E of the degree of conformance of the feature of the element action section before and after the boundary j to the probability model of that element action section with respect to time ｊ Then, this device calculates the time t corresponding to the boundary j as the reference time T ｊ When changed to E ｊ Time width where (t) falls within a predetermined range || t-T ｊ The larger the maximum value of ||, the lower the reliability of the estimation result of the segmentation of the element action section is evaluated.

[0004] In addition, for example, a work element time output device has been proposed that acquires work status information or work position information using various sensors, estimates work breaks from the acquired information, and calculates, records, and outputs work element times from the estimated work breaks.

[0005] International Publication No. WO 2023 / 105788 International Publication No. WO 2023 / 105787 JP 2009-294732 A

[0006] The device for detecting or evaluating the above-mentioned behavioral intervals detects or evaluates behavioral intervals offline from video footage of a series of actions in which multiple component actions are performed in sequence. However, there are cases in which it is desired to detect the boundaries of component actions in real time when the input video information is not complete at the start of detection and the video information increases over time.

[0007] Furthermore, the above-mentioned task element time output device determines that an action has ended when a specific state defined for a series of input sensor information is detected, making it applicable to real-time detection. However, it is difficult to define the end of an action because there are few characteristics, and if the definition is inappropriate, there is a possibility that a part of an action that is not the end of the action will be mistakenly detected as the end of the action. To improve detection accuracy, it is desirable to delay the detection timing and collect a large amount of observation information, but in this case, the delay between the actual execution of the action and its detection will be large.

[0008] In one aspect, the disclosed technology aims to detect the end time of each component action in a series of actions in which multiple component actions are executed in sequence in real time, while simultaneously improving detection accuracy and reducing delays.

[0009] In one aspect, the disclosed technology extracts observation information from video of a person performing a plurality of component actions in sequence. The disclosed technology defines a first likelihood as a likelihood of first observation information from a start time of the action to a current time for a first partial detection model including a probabilistic model representing each component action from the start of the action to a detection target component action. The disclosed technology defines a second likelihood as a likelihood of the first observation information for a second partial detection model including a probabilistic model representing each component action from the start of the action to a component action subsequent to the detection target component action. If the second likelihood is higher than the first likelihood, the disclosed technology detects the end time of the detection target component action in the action based on the second partial detection model.

[0010] One aspect is that in a series of actions in which multiple component actions are executed in sequence, the end time of each component action can be detected in real time, while improving detection accuracy and reducing delays.

[0011] 1 is a diagram for explaining offline detection of end times of element actions. FIG. 2 is a diagram for explaining detection of end times of element actions using a partial detection model. FIG. 3 is a functional block diagram of an action section detection device. FIG. 4 is a conceptual diagram of a hidden semi-Markov model, which is an example of a partial detection model. FIG. 5 is a conceptual diagram illustrating a state of a first hidden Markov model. FIG. 6 is a diagram for explaining detection of end times of element actions in the first embodiment. FIG. 7 is a diagram for explaining detection of end times of element actions in the first embodiment. FIG. 8 is a diagram for explaining detection of end times of element actions in the first embodiment. FIG. 9 is a block diagram showing a schematic configuration of a computer functioning as an action section detection device. FIG. 10 is a flowchart showing an example of machine learning processing. FIG. 11 is a flowchart showing an example of detection processing. FIG. 12 is a diagram for explaining detection of end times of element actions in the second embodiment.

[0012] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0013] As shown in FIG. 1, the feature quantity extracted from the observation information observed at time t (t=1, 2, . . . , T) is expressed as x ｔ In this case, the time series data of the feature is X = {x １ , x ２ , ..., x Ｔ}. A detection model representing a series of behaviors using a probabilistic model representing the i-th (i=1, 2, ..., N) element behavior (hereinafter referred to as "element behavior i") in a series of behaviors in which a plurality of element behaviors are executed in order is expressed as M １：Ｎ In this case, the time series data X of the feature quantity and the detection model M １：Ｎ and assigning times 1 to T to each of the elemental actions so that the likelihood of the probability model for each elemental action is maximized, thereby making it possible to detect the time interval for each elemental action.

[0014] In this case, a partial detection model M １：ｎ Similarly, the feature time series data X and the partial detection model M １：ｎ For example, as shown in FIG. 2, at time t=6, the partial detection model M １：１ Then, at time t=9, the partial detection model M １：２ The end time of the element action section of element action 2 is detected using

[0015] In this way, the part detection model M １：ｎ By utilizing the partial detection model M, it is possible to detect the end time of each element action in real time. １：ｎ When detecting the end time of each element behavior by sequentially applying the above methods, the end time may not be detected accurately due to variations in the observation information at the time of detection compared to the observation information during machine learning of the partial detection model. Each of the following embodiments provides a detection method that suppresses a decrease in detection accuracy while suppressing delays in real-time detection. Each embodiment will be described in detail below.

[0016] 3 , the action section detection device 10 according to the first embodiment functionally includes an extraction unit 11, a machine learning unit 20, and an element action section detection unit 30. The machine learning unit 20 further includes an observation probability learning unit 21, a transition probability learning unit 22, and a construction unit 23. The element action section detection unit 30 further includes a calculation unit 31 and a detection unit 32. A partial detection model 41 is stored in a predetermined storage area of ​​the action section detection device 10.

[0017] The extraction unit 11 acquires learning video during machine learning. The learning video is video of a person's actions, to which the end time of each element action included in a series of actions is added as training information. The time period separated by the end time is the element action interval of each element action, and the element action interval from the first element action 1 to the last element action N is the action interval that indicates the time interval of the series of actions. The extraction unit 11 extracts time-series data of feature amounts related to human movements from the video of the action interval of the learning video.

[0018] Furthermore, the extraction unit 11 acquires detection video from time to time during detection. The detection video is video of a person's behavior, and the end time of each elemental behavior is unknown. In this embodiment, the start time of the detection video is the start time of the first elemental behavior section, or the start time of the first elemental behavior section is assigned to the detection video. The start time of the first elemental behavior is detected separately, for example, by input information from a start button pressed by the worker, information read by a barcode by the worker, or the like.

[0019] An example of a method for extracting time-series data of feature quantities from video by the extraction unit 11 will be described in detail. The extraction unit 11 applies a person detection technology such as YOLO (You Only Look Once) to detect human regions (e.g., bounding boxes) from each frame constituting a video (learning video or detection video), and tracks the detected regions of the same person by associating them between frames. When multiple human regions are detected from a single frame, the extraction unit 11 identifies the region of the person to be determined based on the size of the region, the position of the region within the frame, etc. The extraction unit 11 performs image processing on images within the human regions detected from each frame and calculates posture information based on the positions of the person's joints and their connection relationships, etc. The extraction unit 11 creates time-series posture information by associating the posture information calculated for each frame with time information associated with the frame and arranging them.

[0020] The extraction unit 11 also calculates time-series motion information for each body part from the time-series posture information. The motion information may be, for example, the degree of bending of each part, the speed of bending, etc. Each part may be, for example, an elbow, a knee, etc. The extraction unit 11 also calculates a feature vector whose elements are values ​​obtained by averaging the motion information within the window in the time direction at regular time intervals using a sliding time window. The extraction unit 11 converts the feature vector calculated from the frame at time t into a feature x ｔ Let's say.

[0021] During machine learning, the extraction unit 11 passes the time-series data of the extracted features and supervised information indicating the end times of each elemental action in the learning video as supervised data to the machine learning unit 20. During detection, the extraction unit 11 passes the time-series data of the extracted features to the elemental action section detection unit 30.

[0022] The machine learning unit 20 generates a part detection model 41 by performing machine learning using the supervised data passed from the extraction unit 11 .

[0023] In this embodiment, a hidden semi-Markov model (hereinafter referred to as an HSMM) as shown in Fig. 4 is constructed as an example of a partial detection model 41 for detecting the end time of each elemental action. The HSMM has a probability distribution of the duration of each state as a parameter in addition to the parameters of the hidden Markov model (hereinafter referred to as an HMM).

[0024] The HSMM of this embodiment includes a plurality of first HMMs that represent each of human actions as a state, and a second HMM that represents elemental actions as states. In FIG. 4 , m1, m2, and m3 are states corresponding to each action, and a1, a2, and a3 are states corresponding to each elemental action. An elemental action is a combination of multiple actions, and an action is a combination of multiple postures. When time-series data on features related to human actions extracted from video is provided to the HSMM constructed by setting parameters, the HSMM detects optimal elemental action interval boundaries. In FIG. 4 , d1, d2, and d3 are examples of elemental action intervals.

[0025] HMM parameters include observation probabilities and transition probabilities. In FIG. 4, O1, O2, ..., O8 are examples of observation probabilities. Transition probabilities are also associated with arrows connecting states. The observation probability is the probability that certain observation information is observed in each state, and the transition probability is the probability of transitioning from one state to another. If the order of transitions is fixed, transition probabilities are not necessary. Note that the number of actions and the number of component actions, i.e., the number of first HMMs and second HMMs, are merely examples and are not limited to the numbers exemplified in FIG. 4. The observation probability learning unit 21, transition probability learning unit 22, and construction unit 23 included in the machine learning unit 20 will be described in detail below.

[0026] As described below, the observation probability learning unit 21 learns the observation probability of each action constituting the HSMM, which is an example of the partial detection model 41, using time-series data of features obtained by removing the supervised information from the supervised data (hereinafter also referred to as "unsupervised data").

[0027] In this embodiment, the target behavior is a limited behavior for achieving a certain task goal. Such behavior is, for example, the behavior of a person manufacturing a product, such as the behavior of routine work performed on a factory line, and has the following properties: Property 1: The differences between the component actions that make up the behavior are differences in the combination of multiple limited movements. Property 2: Multiple postures observed when performing the same behavior are similar.

[0028] In this embodiment, based on property 1, all element actions are composed of actions included in one action group. For example, as shown in FIG. 5 , the action group includes three actions m11, m12, and m13. For example, action m11 may be "raising the arm," action m12 may be "lowering the arm," and action m13 may be "stretching the arm forward." In other words, element actions are combinations of human actions when performing each manufacturing process. The number of actions included in an action group is not limited to the example of FIG. 5 . Furthermore, the number of actions included in each element action is not limited to the example of FIG. 5 .

[0029] For example, the observation probability learning unit 21 calculates the observation probability of each action using a Gaussian Mixture Model (hereinafter referred to as "GMM (Gaussian Mixture Model)"). Specifically, the observation probability learning unit 21 estimates parameters of a GMM in which as many Gaussian distributions as there are actions are mixed by clustering the feature amounts passed from the extraction unit 11. Then, the observation probability learning unit 21 assigns each Gaussian distribution constituting the GMM whose parameters have been estimated as a probability distribution representing the observation probability of each action.

[0030] As will be described below, the transition probability learning unit 22 calculates the transition probabilities between actions represented by the first HMM based on supervised data. Specifically, the transition probability learning unit 22 allocates the time-series data of the feature amounts to each element action interval based on the supervised information contained in the supervised data. The transition probability learning unit 22 then uses the time-series data of the feature amounts allocated to each element action interval as observation information, fixes the observation probability of each action calculated by the observation probability learning unit 21, and calculates the transition probabilities between actions using, for example, maximum likelihood estimation or an Expectation-Maximization algorithm (EM algorithm).

[0031] Since creating supervised data takes time and effort, the transition probability learning unit 22 may pad the supervised data by adding noise to the seed supervised data, for example.

[0032] The construction unit 23 sets a probability distribution of the duration of each element action based on the duration of each element action interval given in the teacher information. For example, the construction unit 23 sets a uniform distribution within a predetermined range for the duration of each element action interval given in the teacher information as the probability distribution of the duration of the element action.

[0033] The construction unit 23 constructs an HSMM, such as that shown in FIG. 4, as a partial detection model 41 using the observation probability of each action calculated by the observation probability learning unit 21, the transition probability between actions calculated by the transition probability learning unit 22, and the duration of each set element action. The partial detection model 41 is an HSMM in which the second HMM corresponding to each element action transitions in the order of the element actions given in the teacher information after the set duration. In FIG. 4, O1, O2, ..., O8 represent observation probabilities calculated by the observation probability learning unit 21. Furthermore, the transition probabilities associated with the arrows between actions m1, m2, and m3 included in each element action a1, a2, and a3 correspond to the transition probabilities calculated by the transition probability learning unit 22. Furthermore, d1, d2, and d3 represent the duration of each element action.

[0034] In this embodiment, the construction unit 23 constructs a partial detection model M including a trained second HMM (probabilistic model) representing each of the element actions from the first element action 1 to element action n. １：ｎ The constructing unit 23 constructs the part detection model M for all cases where n is 1 to N. １：ｎ The constructing unit 23 constructs the N part detection models M １：ｎ (n=1, 2, ..., N) are stored in a predetermined storage area. In the following, when the N partial detection models are described without distinction, they will be referred to as "partial detection model 41", and when the N partial detection models are described without distinction, they will be referred to as "partial detection model M １：ｎ (n=1, 2, ..., N)".

[0035] The element action interval detection unit 30 detects the end time of each element action interval in the detection video based on the time series data of the feature amount passed from the extraction unit 11. Below, the calculation unit 31 and the detection unit 32 included in the element action interval detection unit 30 will be described in detail.

[0036] The calculation unit 31 calculates a first likelihood of first observation information from the start time of the behavior to the current time for a first partial detection model including a probability model representing each element behavior from the start of the behavior to the element behavior to be detected. The calculation unit 31 also calculates a second likelihood of the first observation information for a second partial detection model including a probability model representing each element behavior from the start of the behavior to an element behavior after the element behavior to be detected. The likelihood is the probability that observation information will be observed from the probability model representing each element behavior included in the partial detection model 41.

[0037] Specifically, the calculation unit 31 sets the element behavior to be detected as the n-th element behavior, and calculates a partial detection model M corresponding to each element behavior from the start of the behavior to the element behavior n. １：ｎ Based on this, likelihood P(X|M １：ｎ ) corresponding to each element action from the start of the action to element action n+α (1<α<N). １：ｎ＋α Based on this, the likelihood P(X|M １：ｎ＋α ) is calculated.

[0038] For example, Fig. 6 shows an example of calculating the likelihood when n = 1, α = 1, and the current time is t = 6. As shown in Fig. 6, the calculation unit 31 calculates the likelihood by using the part detection model M １：１ Observation information X = {x １ , x ２ , ..., x ６} likelihood P(X|M １：１ ) is calculated. The calculation unit 31 also calculates the part detection model M １：２ Observation information X = {x １ , x ２ , ..., x ６} likelihood P(X|M １：２ ) is calculated. When the partial detection model 41 includes two or more probability models, the calculation unit 31 allocates the time 1 to t to each element behavior based on the probability distribution of the duration of the element behavior, and compares the probability model of each element behavior with the observation information. In the example of FIG. 6, the calculation unit 31 calculates the partial detection model M １：２ Among them, the feature x for the probability model of element behavior 1 １ ~x ４Calculate the likelihood of the feature x ５ ~x ６ The likelihoods of the two are calculated and multiplied to obtain the likelihood P(X|M １：２ ) is calculated.

[0039] When the second likelihood is higher than the first likelihood, the detection unit 32 detects the end time of the element action to be detected in the series of actions based on the second partial detection model. In the example of FIG. 6, the detection unit 32 １：２ ) > P(X | M １：１ ), t=4 is detected as the end time of component action 1.

[0040] The calculation unit 31 and the detection unit 32 repeat the detection of the end time of the component action n from n=2 onwards. For example, as shown in FIG. 6, the end time (t=4) of the component action 1 is detected, and the end time of the component action 2 is not detected at t=7 and t=8, and the time reaches t=9. FIG. 7 shows a case where the calculation unit 31 detects the end time of the component action n using the partial detection model M １：２ Observation information X = {x １ , x ２ , ..., x ９} likelihood P(X|M １：２ ) is calculated. The calculation unit 31 also calculates the part detection model M １：３ Observation information X = {x １ , x ２ , ..., x ９} likelihood P(X|M １：３ Then, the detection unit 32 calculates P(X|M １：３ ) > P(X | M １：２ ), t=7 is detected as the end time of component action 2.

[0041] 6 and 7, the case where α=1 has been described, but even if α is 2 or more, the end time of the element behavior to be detected can be detected in the same way. For example, FIG. 8 shows an example where n=1, α=2, and the current time is t=6. In this case, the calculation unit 31 calculates the partial detection model M １：１ Observation information X = {x １ , x ２ , ..., x ６} likelihood P(X|M １：１) is calculated. The calculation unit 31 also calculates the part detection model M １：３ Observation information X = {x １ , x ２ , ..., x ６} likelihood P(X|M １：３ Then, the detection unit 32 calculates P(X|M １：３ ) > P(X | M １：１ ), t=3 is detected as the end time of component action 1.

[0042] As described above, the likelihood of the observation information for the partial detection model 41 is the probability that the partial detection model 41 will generate time-series data of feature quantities up to the current time, which is the observation information. However, the acquired observation information varies each time it is acquired compared to the observation information used as training data. This variation in the observation information also causes variation in the calculated likelihood. Therefore, when detecting the end time of an element behavior simply based on the likelihood for the partial detection model 41 up to the element behavior to be detected, the detection accuracy may decrease. Furthermore, when the duration of each probability model included in the partial detection model 41 is used, the required time for each element behavior for each behavior varies, so the detection accuracy may also decrease as described above.

[0043] In this embodiment, as described above, in detecting the end time of the element behavior n, the partial detection model M １：ｎ Likelihood of observation information P(X|M １：ｎ ) and the part detection model M １：ｎ＋α Likelihood of observation information P(X|M １：ｎ＋α ) is compared with P(X|M １：ｎ＋α ) > P(X | M １：ｎ ), it is assumed that the series of actions has progressed up to element action n+α. If element action n+α has progressed, it is assumed that the end of element action n is at least confirmed, and the end time of element action n is detected. This makes it possible to prevent a decrease in detection accuracy due to variations in observation information. Note that P(X|M １：ｎ＋α ) > P(X | M １：ｎ), the end time of the component behavior n+α is not determined at this stage, so the end time of the component behavior n+α is not detected.

[0044] Furthermore, when the last element action in a series of actions is N, all or some of the element actions after element action N-α up to element action n+α do not exist. Therefore, when element actions after element action N-α are to be detected, the calculation unit 31 and the detection unit 32 detect the end times of the element actions as follows.

[0045] The calculation unit 31 calculates the relative conformance of the second observation information from the end time of the detected component behavior N-α to the current time at each time after the end time of the component behavior N-α. The relative conformance is a value obtained by dividing the likelihood of the second observation information with respect to the probabilistic model of component behavior n by the likelihood of the second observation information with respect to the probabilistic models a1 to aN representing each of component behaviors 1 to N.

[0046] 9, a case will be described in which n=N and α=1, the end time T-3 of the component behavior N-1 has already been detected, and the current time is t=T. If the second observation information is X', then in this case, X'={x Ｔ－２ , x Ｔ－１ , x Ｔ The calculation unit 31 calculates the relative fitness E of the component behavior n at time t using the following formula (1): ｔ Calculate E ｔ =P(X'|an) / Σ ｉ P(X'|ai) (1) P(X'|an) is the likelihood of the observed information X' for the probabilistic model an (here, n = N) of the element behavior to be detected, and P(X'|ai) is the likelihood of the observed information X' for the probabilistic model ai (i = 1, 2, ..., N) of the element behavior i.

[0047] The detection unit 32 detects the time when the relative fitness calculated by the calculation unit 31 is at a maximum as the end time of the element behavior n to be detected. ｔ When the value of the element behavior N reaches a maximum, the detection unit 32 detects the time T as the end time of the element behavior N.

[0048] The detection unit 32 outputs the detected end time as a detection result every time it detects the end time of each component action n (n=1, 2, ..., N). In this way, the boundaries of component actions are detected in real time.

[0049] The action interval detection device 10 may be realized by, for example, a computer 50 shown in FIG. 10 . The computer 50 includes a CPU (Central Processing Unit) 51, a GPU (Graphics Processing Unit) 52, a memory 53 serving as a temporary storage area, and a non-volatile storage device 54. The computer 50 also includes an input / output device 55 such as an input device and a display device, and an R / W (Read / Write) device 56 that controls reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I / F (Interface) 57 that is connected to a network such as the Internet. The CPU 51, GPU 52, memory 53, storage device 54, input / output device 55, R / W device 56, and communication I / F 57 are connected to one another via a bus 58.

[0050] The storage device 54 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 54 serving as a storage medium stores an action interval detection program 60 for causing the computer 50 to function as the action interval detection device 10. The action interval detection program 60 includes an extraction process control instruction 61, a machine learning process control instruction 70, and an element action interval detection process control instruction 80. The storage device 54 also includes an information storage area 90 in which information constituting the partial detection model 41 is stored.

[0051] The CPU 51 reads the action interval detection program 60 from the storage device 54, loads it into the memory 53, and sequentially executes the processes of the action interval detection program 60. The CPU 51 operates as the extraction unit 11 shown in FIG. 3 by executing the extraction process control instruction 61. The CPU 51 also operates as the machine learning unit 20 shown in FIG. 3 by executing the machine learning process control instruction 70. The CPU 51 also operates as the element action interval detection unit 30 shown in FIG. 3 by executing the element action interval detection process control instruction 80. The CPU 51 also reads information from the information storage area 90 and loads the partial detection model 41 into the memory 53. As a result, the computer 50 that executed the action interval detection program 60 functions as the action interval detection device 10. The CPU 51 that executes the program is hardware. A portion of the program may be executed by the GPU 42.

[0052] The functions realized by the action interval detection program 60 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0053] Next, the operation of the action section detection device 10 according to the first embodiment will be described. When a learning video is input to the action section detection device 10 and an instruction is given to execute machine learning of the part detection model 41, the action section detection device 10 executes the machine learning process shown in Fig. 11. Furthermore, when a detection video is input to the action section detection device 10 and an instruction is given to detect the end time of each element action, the action section detection device 10 executes the detection process shown in Fig. 12. Note that the machine learning process and the detection process are examples of the action section detection method of the disclosed technology.

[0054] First, the machine learning process shown in FIG. 11 will be described.

[0055] In step S11, the extraction unit 11 acquires the learning video input to the action segment detection device 10, and extracts time-series data of feature amounts related to human actions from the learning video.

[0056] Next, in step S12, the observation probability learning unit 21 estimates parameters of a GMM in which Gaussian distributions corresponding to the number of actions are mixed by clustering the features extracted in step S11. Then, the observation probability learning unit 21 assigns each Gaussian distribution constituting the GMM whose parameters have been estimated as a probability distribution representing the observation probability of each action.

[0057] Next, in step S13, the transition probability learning unit 22 allocates the time-series data of the feature amounts extracted in step S11 to each element action interval indicated by the supervised information of the supervised data. Next, in step S14, the transition probability learning unit 22 uses the time-series data of the feature amounts allocated to each element action interval as observation information, fixes the observation probability of each action calculated in step S12, and calculates the transition probability between actions.

[0058] Next, in step S15, the construction unit 23 sets a probability distribution of duration for each element action based on the duration of each element action section given in the teacher information. Next, in step S16, the construction unit 23 constructs an HSMM based on the observation probability of each action calculated in step S12, the transition probability between actions calculated in step S14, and the duration of each element action set in step S15, as a partial detection model 41. Here, the construction unit 23 constructs a partial detection model M including a trained second HMM (probability model) representing each element action from the first element action 1 to element action n. １：ｎ The constructing unit 23 constructs the part detection model M for all cases where n is 1 to N. １：ｎ The construction unit 23 constructs the constructed part detection model M １：ｎ (n=1, 2, . . . , N) is stored in a predetermined storage area, and the machine learning process ends.

[0059] Next, the detection process shown in FIG. 12 will be described.

[0060] In step S21, the detection unit 32 sets a variable n that identifies the element behavior to be detected to 1. Next, in step S22, the extraction unit 11 acquires a frame at time t, which corresponds to the start time of the first element behavior, in the video input to the behavior section detection device 10, and extracts a feature quantity x ｔ Extract.

[0061] Next, in step S23, the detection unit 32 determines whether n is equal to or less than N-α, which is the value obtained by subtracting a constant α (1<α<N) from the number of element actions N. If n≦N-α, the process proceeds to step S24, and if n>N-α, the process proceeds to step S28.

[0062] In step S24, the calculation unit 31 calculates a partial detection model M corresponding to each element action from the start of the action to the element action n. １：ｎ Based on this, the likelihood P(X|M １：ｎ ) corresponding to each element action from the start of the action to element action n+α. １：ｎ＋α Based on this, the likelihood P(X|M １：ｎ＋α ) is calculated.

[0063] Next, in step S25, the detection unit 32 calculates P(X|M １：ｎ ) than P(X|M １：ｎ＋α ) is higher than P(X|M １：ｎ＋α ) > P(X | M １：ｎ ), the process proceeds to step S26. １：ｎ＋α ) ≦ P(X|M １：ｎ ), the process returns to step S22 and repeats the process for the next frame at time t.

[0064] In step S26, the detection unit 32 detects the part detection model M １：ｎ＋α Then, in step S27, the detecting unit 32 increments n by 1, and the process returns to step S22.

[0065] In step S28, the calculation unit 31 calculates the relative fitness E of the observation information X' from the end time of the detected component behavior N-α to the current time, for example, using equation (1). ｔ Next, in step S29, the detection unit 32 calculates the relative compatibility E ｔ Determine whether the relative fitness E ｔ If it has reached a maximum, the process proceeds to step S30, and if it has not reached a maximum, the process returns to step S22.

[0066] In step S30, the detection unit 32 calculates the relative compatibility E ｔ The time when n reaches a maximum is detected as the end time of the component behavior n. Next, in step S31, the detection unit 32 determines whether n has become N. If n<N, the process proceeds to step S32, where the detection unit 32 increments n by 1, and returns to step S22. If n=N, the detection process ends.

[0067] As described above, the action section detection device according to the first embodiment extracts time-series data of feature quantities, which are observation information, from video footage of a person sequentially performing multiple component actions. The action section detection device also calculates a first likelihood of first observation information from the start time of the action to the current time for a first partial detection model including a probabilistic model representing each component action from the start of the action to the component action to be detected. The action section detection device also calculates a second likelihood of the first observation information for a second partial detection model including a probabilistic model representing each component action from the start of the action to the component action after the component action to be detected. If the second likelihood is higher than the first likelihood, the action section detection device detects the end time of the component action to be detected in the action based on the second partial detection model. This makes it possible to detect the end time of each component action in a series of actions in which multiple component actions are performed sequentially in real time, while achieving both improved detection accuracy and reduced delays.

[0068] In the first embodiment, the case where α is fixed has been described, but the value of α may be varied for each elemental behavior to be detected depending on the order, duration, etc. of the elemental behavior to be detected.

[0069] Second Embodiment Next, a second embodiment will be described. In the activity section detection device according to the second embodiment, components similar to those in the activity section detection device 10 according to the first embodiment will be assigned the same reference numerals, and detailed descriptions thereof will be omitted. Components having some of the same functions will be assigned reference numerals having the same last two digits, and detailed descriptions of the common parts will be omitted.

[0070] 3 , the action interval detection device 210 according to the second embodiment functionally includes an extraction unit 11, a machine learning unit 20, and an element action interval detection unit 230. The element action interval detection unit 230 further includes a calculation unit 231 and a detection unit 232. A partial detection model 41 is stored in a predetermined storage area of ​​the action interval detection device 210.

[0071] As described above, the observation information at the time of detection varies with respect to the observation information used for machine learning of the partial detection model 41, and therefore the likelihood of the probability model for each elemental behavior included in the partial detection model 41 may not be what is expected from the progress of the behavior. For example, simply assuming n+α=n+1, the partial detection model M up to the elemental behavior to be detected may be １：ｎ and the partial detection model M １：ｎ＋１ In this case, even though the next element action n to be detected has not yet ended, P(X|M １：ｎ＋１ ) > P(X | M １：ｎ On the other hand, if the observed information of component action n+1 at the time of detection is significantly different from the time of machine learning of the probability model of component action n+1, then even if component action n+1 starts, P(X|M １：ｎ＋１ ) > P(X | M １：ｎ ) may not hold. In such a case, P(X|M １：ｎ＋１ ) and P(X|M １：ｎ ) may not be able to properly detect the end time of the element behavior n.

[0072] The second embodiment provides a detection method that can handle cases where the observed information at the time of detection is partially significantly deviated from that at the time of machine learning of the partial detection model 41. The calculation unit 231 and the detection unit 232 of the second embodiment will be described in detail below.

[0073] The calculation unit 231 calculates the third likelihood of the first observation information from the start time of the action to the current time for each of all the part detection models 41 corresponding to each element action from the start of the action to each turn. That is, the calculation unit 231 calculates the third likelihood of the first observation information X={x1, ..., x ｔ}, all part detection models M １：ｎ (n=1, 2, ..., N) １：ｎ ) is calculated.

[0074] When the partial detection model 41 with the maximum third likelihood includes probability models up to the element action i, the detection unit 232 determines whether i is greater than n, i.e., whether the element action i is ordered after the element action n. If i > n, the detection unit 232 sets the partial detection model 41 with the maximum third likelihood as the second partial detection model in the first embodiment. Then, when the ratio or difference of the second likelihood to the first likelihood is greater than a predetermined value, the detection unit 232 detects the end time of the element action n based on the second partial detection model.

[0075] Specifically, the detection unit 232 calculates each part detection model M calculated by the calculation unit 231 using the following formula (2): １：ｎ The likelihood P(X|M １：ｎ ) is the order of element actions n ｍａｘ Identify.

[0076]

[0077] The detection unit 232 detects n ｍａｘ >n and P(X|M １：ｎ_ｍａｘ ) / P(X | M １：ｎ )>C, the part detection model P(X|M １：ｎ_ｍａｘ ) to detect the end time of element behavior n. C is a constant and may be an experimentally determined value. The latter is used as a condition for the timing of detecting the end time because detection is performed when the likelihood ratio is above a certain level, taking into account the likelihood due to variations in the observation information at the time of detection as described above. Note that P(X|M １：ｎ_ｍａｘ ) / P(X | M １：ｎ )>C, and P(X|M１：ｎ_ｍａｘ ) -P(X|M １：ｎ ) > C may also be used.

[0078] 13 shows an example in which n=1. For example, each part detection model M １：ｎ The likelihood P(X|M １：ｎ ) from the maximum likelihood of ｍａｘ In this case, n = 2 is specified. ｍａｘ > n. Therefore, P(X|M １：２ ) / P(X | M １：１ )>C, the part detection model M １：２ The end time of the component action 1 is detected based on the above.

[0079] The action interval detection device 210 may be realized by, for example, a computer 50 shown in Fig. 10. A storage device 54 of the computer 50 stores an action interval detection program 260 for causing the computer 50 to function as the action interval detection device 210. The action interval detection program 260 includes an extraction process control instruction 61, a machine learning process control instruction 70, and an element action interval detection process control instruction 280. The storage device 54 also includes an information storage area 90 for storing information constituting the partial detection model 41.

[0080] The CPU 51 reads the action interval detection program 260 from the storage device 54, loads it into the memory 53, and sequentially executes the processes of the action interval detection program 260. The CPU 51 operates as the element action interval detection unit 230 shown in FIG. 3 by executing the element action interval detection process control instruction 280. Other control instructions are the same as those of the action interval detection program 60 according to the first embodiment. As a result, the computer 50 that has executed the action interval detection program 260 functions as the action interval detection device 210. The CPU 51 that executes the program is hardware. Furthermore, part of the program may be executed by the GPU 42.

[0081] The functions realized by the action interval detection program 260 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC, an FPGA, or the like.

[0082] Next, the operation of the action interval detection device 210 according to the second embodiment will be described. In the second embodiment, the machine learning process shown in FIG. 11 and the detection process shown in FIG. 12 are also executed. In the detection process, in step S24, the calculation unit 231 calculates the machine learning process for all the part detection models M １：ｎ (n=1, 2, ..., N) １：ｎ In step S25, the detection unit 232 calculates n ｍａｘ >n and P(X|M １：ｎ_ｍａｘ ) / P(X | M １：ｎ )>C. In step S26, the detection unit 232 determines whether the partial detection model P(X|M １：ｎ_ｍａｘ ) to detect the end time of the element behavior n.

[0083] As described above, the action section detection device according to the second embodiment selects a part detection model M that maximizes the likelihood of observation information for each part detection model. １：ｎ＿ｍａｘ , n ｍａｘ >n, the part detection model M １：ｎ＋α This makes it possible to stably detect the end time of each element action even if the observed information at the time of detection is partially deviated significantly, compared to when using machine learning with a partial detection model.

[0084] In the above embodiments, the case where the part detection model is HSMM has been described, but this is not limited to this, and other machine learning models such as a model using a neural network may also be applied.

[0085] Furthermore, in each of the above embodiments, the case where the action section detection device including the machine learning unit and the element action section detection unit is realized by one computer has been described, but each may also be realized by a separate computer.

[0086] In addition, although the above-described embodiments each include a case in which the behavioral section detection program is stored (installed) in advance in a storage device, the present invention is not limited to this. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0087] 10, 210 Action interval detection device 11 Extraction unit 20 Machine learning unit 21 Observation probability learning unit 22 Transition probability learning unit 23 Construction unit 30, 230 Element action interval detection unit 31, 231 Calculation unit 32, 232 Detection unit 41 Partial detection model 50 Computer 51 CPU 52 GPU 53 Memory 54 Storage device 55 Input / output device 56 R / W device 57 Communication I / F 58 Bus 59 Storage medium 60, 261 Action interval detection program 61 Extraction process control command 70 Machine learning process control command 80, 280 Element action interval detection process control command 90 Information storage area

Claims

1. A behavioral section detection program for causing a computer to execute a process including: extracting observation information from video footage of a person performing a number of elemental actions in sequence; and, when a second likelihood of the first observation information for a second partial detection model including a probability model representing each elemental action from the start of the behavior to the elemental action after the elemental action to be detected is higher than a first likelihood of the first observation information for a first partial detection model including a probability model representing each elemental action from the start of the behavior to the current time, detecting the end time of the elemental action to be detected in the behavior based on the second partial detection model.

2. The behavior section detection program of claim 1, wherein the component behavior to be detected is the nth component behavior, the first likelihood is calculated based on the first partial detection model corresponding to each component behavior from the start of the behavior to the nth component behavior, and the second likelihood is calculated based on the second partial detection model corresponding to each component behavior from the start of the behavior to the n+αth component behavior.

3. The behavior section detection program of claim 1, wherein the program calculates a third likelihood of the first observation information for each of all partial detection models corresponding to each element behavior from the start of the behavior to each order, the element behavior to be detected is the nth element behavior, and when the partial detection model for which the third likelihood is maximum includes a probabilistic model representing element behaviors up to the i-th element behavior, if i is greater than n, the partial detection model for which the third likelihood is maximum is set as the second partial detection model, and if the ratio or difference of the second likelihood to the first likelihood is greater than a predetermined value, the program detects the end time of the n-th element behavior based on the second partial detection model.

4. A behavior section detection program according to any one of claims 1 to 3, wherein when the plurality of elemental actions are N elemental actions and an elemental action after the N-αth elemental action is the nth elemental action to be detected, a relative fitness is calculated at each time after the end time of the N-αth elemental action by dividing the likelihood of second observation information from the end time of the N-αth elemental action to the current time with respect to the probabilistic model of the nth elemental action by the likelihood of said second observation information with respect to the probabilistic model representing each of said plurality of elemental actions, and the time at which said relative fitness is maximum is detected as the end time of said nth elemental action.

5. A behavioral interval detection program according to any one of claims 1 to 3, for causing the computer to execute processing further comprising: training a probabilistic model representing each of the component behaviors using observation information extracted from video in which the end time of each component behavior is known as training information; and constructing a partial detection model including a trained probabilistic model representing each of the component behaviors from the start of the behavior to each turn.

6. A behavioral interval detection program according to any one of claims 1 to 3, wherein the behavior is a behavior of a person manufacturing a product, and the component behavior is a combination of the person's actions when performing each process of the manufacturing.

7. A behavior section detection method in which a computer executes processes including: extracting observation information from video footage of a person performing a number of component behaviors in sequence; and, when a second likelihood of the first observation information for a second partial detection model including a probabilistic model representing each component behavior from the start of the behavior to an component behavior subsequent to the component behavior to be detected is higher than a first likelihood of the first observation information from the start time of the behavior to the current time for a first partial detection model including a probabilistic model representing each component behavior from the start of the behavior to the component behavior to be detected, detecting the end time of the component behavior to be detected in the behavior based on the second partial detection model.

8. A behavior section detection method as described in claim 7, wherein the element behavior to be detected is the nth element behavior, the first likelihood is calculated based on the first partial detection model corresponding to each element behavior from the start of the behavior to the nth element behavior, and the second likelihood is calculated based on the second partial detection model corresponding to each element behavior from the start of the behavior to the n+αth element behavior.

9. A behavior section detection method as described in claim 7, which calculates a third likelihood of the first observation information for each of all partial detection models corresponding to each element behavior from the start of the behavior to each order, sets the element behavior to be detected as the nth element behavior, and when the partial detection model for which the third likelihood is maximum includes a probabilistic model representing element behaviors up to the i-th element behavior, if i is greater than n, sets the partial detection model for which the third likelihood is maximum as the second partial detection model, and detects the end time of the n-th element behavior based on the second partial detection model if the ratio or difference of the second likelihood to the first likelihood is greater than a predetermined value.

10. A behavior section detection method according to any one of claims 7 to 9, wherein the plurality of elemental actions are N elemental actions, and an elemental action after the N-αth elemental action is the nth elemental action to be detected, the method calculates a relative fitness for each time after the end time of the N-αth elemental action by dividing the likelihood of second observation information from the end time of the N-αth elemental action to the current time with respect to a probabilistic model of the nth elemental action by the likelihood of said second observation information with respect to a probabilistic model representing each of said plurality of elemental actions, and detects the time at which said relative fitness is maximum as the end time of said nth elemental action.

11. A behavior section detection method according to any one of claims 7 to 9, for causing the computer to execute processing further comprising: training a probabilistic model representing each of the component behaviors using observation information extracted from video in which the end time of each component behavior is known as training information; and constructing a partial detection model including a trained probabilistic model representing each of the component behaviors from the start of the behavior to each turn.

12. A behavior interval detection method according to any one of claims 7 to 9, wherein the behavior is a behavior of a person manufacturing a product, and the component behavior is a combination of the person's actions when performing each process of the manufacturing.

Images