Action interval evaluation program, device, and method

Abstract

An action interval evaluation device according to the present invention uses, as a reference, a time Tj corresponding to a boundary j in an estimation result for segmentation of element action intervals based on time-series feature amounts which are extracted from a video that is obtained by capturing an action of a person and on an estimation model which includes respective probability models of a plurality of element actions included in the action, and calculates a function Ej(t) of a degree of matching of feature amounts, when the boundary j is shifted to various times, of element action intervals before and after the boundary j to a probability model of said element action intervals, with respect to the time. When a time t corresponding to the boundary j is changed with respect to the reference time Tj, the action interval evaluation device evaluates the estimation result for the segmentation of the element action intervals to be less reliable as the maximum value of a time width ||t-Tj|| in which Ej(t) is in a prescribed range increases.

Description

[Technical Field]

[0001] The disclosed technology relates to an action interval evaluation program, an action interval evaluation device, and an action interval evaluation method. [Background technology]

[0002] Advances in deep learning technology have made it possible to recognize human posture and other characteristics with high accuracy from images captured with a standard RGB camera, and various research and development projects are being conducted to utilize this recognition information to estimate human behavior, such as specific tasks.

[0003] For example, a task analysis system has been proposed that estimates a worker's tasks based on task information and calculates a reliability indicating the accuracy of the estimated tasks. This system calculates the start and end times of each estimated task based on time information. This system also divides the acquired task video at the start and end times of the estimated task, and associates the segment video from the start time to the end time of the estimated task with the estimated task and the reliability of the task.

[0004] Furthermore, for example, a work analysis device has been proposed in which, for each unit task within the imaging range of the imaging unit, a first monitoring area for detecting the start of the unit task and a second monitoring area for detecting the completion of the unit task are set. In this device, a reliability is set for each monitoring area such that the higher the probability that it is determined that an action related to the unit task is being performed in that monitoring area, the higher the reliability. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2020-91801 [Patent Document 2] Japanese Patent Application Publication No. 2020-166471 Summary of the Invention [Problem to be solved by the invention]

[0006] Based on time-series features such as posture extracted from a video of a person, the time interval in which a specified action occurred in the video is estimated, and this time interval is further segmented into time intervals of elemental actions included in the specified action. In this case, even if the boundaries between elemental actions deviate from the actual boundaries, the reliability of the segmentation of elemental actions may be evaluated as high.

[0007] In one aspect, the disclosed technology aims to accurately evaluate the reliability of segments of specified actions in a video of a person. [Means for solving the problem]

[0008] In one aspect, the disclosed technology obtains an estimation result of segmentation of an elemental action section, which is a time section of the elemental action, based on time-series feature amounts extracted from a video of a human action and an estimation model including a probabilistic model for each of a plurality of elemental actions included in the action. The disclosed technology also calculates the conformance of the time-series feature amounts corresponding to each of the elemental action sections before and after the boundary between adjacent elemental action sections when the boundary between the elemental action sections is changed to the probabilistic model corresponding to the elemental action. The disclosed technology then evaluates the estimation result of segmentation of the elemental action section based on a change in the conformance depending on the degree of change in the boundary. [Effects of the Invention]

[0009] As one aspect, it has an effect of being able to accurately evaluate the reliability of segments of a specified action in a video of a person. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 10 is a diagram for explaining a comparative example of the present embodiment. [Figure 2] FIG. 10 is a diagram for explaining another comparative example of the present embodiment. [Figure 3] FIG. 10 is a diagram for explaining another comparative example of the present embodiment. [Figure 4] FIG. 10 is a diagram for explaining a problem in the comparative example. [Figure 5] FIG. 2 is a functional block diagram of the action section evaluation device. [Figure 6] FIG. 1 is a conceptual diagram of a hidden semi-Markov model, which is an example of an estimation model. [Figure 7] FIG. 2 is a conceptual diagram illustrating the state of a first hidden Markov model. [Figure 8] 10A and 10B are diagrams for explaining changes in the degree of conformance of feature quantities of element action intervals before and after a boundary to a probabilistic model when the boundary is shifted. [Figure 9] 10 is a graph showing an example of the relationship between time t and the goodness of fit Ej(t) of the boundary j. [Figure 10] 10 is a graph showing an example of the relationship between time t and the goodness of fit Ej(t) of the boundary j. [Figure 11] FIG. 10 is a diagram for explaining the maximum value of the time width. [Figure 12] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as an action section evaluation device. [Figure 13] 10 is a flowchart illustrating an example of machine learning processing. [Figure 14] 10 is a flowchart illustrating an example of segmentation processing. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0012] The action section evaluation device according to this embodiment segments a time section (hereinafter referred to as an "action section") in which a target action detected from a video occurred into component actions based on time-series features of human actions extracted from the video of the person. That is, the action section evaluation device estimates the segmentation of time sections of component actions (hereinafter referred to as "component action sections") within an action section. In this embodiment, for example, the target action is an action in which a person manufactures a product, and the component actions are combinations of human actions when performing each manufacturing process. That is, the target action is an action that includes multiple component actions whose order of occurrence is restricted, such as work in a factory that involves multiple processes in a fixed order. Furthermore, the action section evaluation device evaluates the reliability of the segmentation of the estimated component action sections.

[0013] As a comparative example to this embodiment, a method of manually segmenting video into sections to identify element action sections from video is conceivable. This comparative example is a method of acquiring a video of a series of tasks captured by a camera, as shown in the left diagram of FIG. 1, and manually segmenting the acquired video into element action sections by visually viewing the acquired video, as shown in the right diagram of FIG. 1. In the example of FIG. 1, "inserting part A," "screwing part A," and "attaching the cover" are each examples of element actions. In this way, manually segmenting each acquired video into element action sections requires time and effort.

[0014] As another comparative example of this embodiment, as shown in the upper diagram of Fig. 2, it is also possible to manually segment element action intervals for one video, and then use this segmentation result as teaching information to automatically segment element action intervals for other videos, as shown in the lower diagram of Fig. 2. In this case, it is possible to save time and effort compared to manually segmenting element action intervals for all videos.

[0015] As shown in the example in Figure 2, when automatically segmenting elemental activity intervals using training information, even if the human actions indicated by the training information do not match the human actions appearing in the target video, the boundaries of all elemental activity intervals are output in the same manner as the training information. While the boundaries of elemental activity intervals in time periods that do not match the training information are likely to be incorrect, it is not possible to determine which elemental activity does not match the training information from the estimated results of segmentation of elemental activity intervals alone. Therefore, it is desirable to evaluate the reliability of the segmentation results and output this separately from the estimated elemental activity interval segmentation results. This reliability may not necessarily coincide with the accuracy of the estimated results of segmentation of elemental activity intervals. For example, even time periods that do not match the training information may be correctly segmented based on the context. In this case, the accuracy of the estimated segmentation results is high, but its reliability is low, unlike when the boundaries match the training information and are clear.

[0016] As another comparative example of this embodiment, it is possible to use the relative fitness expressed as the ratio of the likelihood (observation probability) according to the probability model of the estimated element behavior to the likelihood according to the probability model of another element behavior as the reliability for the observed information of the estimated element behavior section. The larger the relative fitness, the more likely the estimation result is to be correct. For example, for observed information X within element behavior section j of element behavior j, the probability model M of each element behavior is used as the reliability. i The likelihood P(X|M i ) and calculate the reliability C of element action j. j It is conceivable to calculate this using the following formula (1): Note that the observation information is a feature amount such as the video or the posture information of a person extracted from the video. C j =P(X|M j ) / Σ i P(X|M i ) ···(1)

[0017] For example, in the comparative example, as shown in FIG. 3, the feature quantities of the time series included in the action section (x1, x2, . . . , x 10) based on an estimation model that models the duration of each elemental action interval, the elemental action intervals are estimated. In Figure 3, the interval of feature quantities x1 to x3 corresponds to elemental action A, the interval of feature quantities x4 to x8 corresponds to elemental action B, and the interval of feature quantities x9 to x 10 is estimated as the element action interval corresponding to element action C. In the comparative example, for each element action interval, the likelihood is calculated for each probability model of all element action intervals for the feature values ​​assigned to that element action interval, and the relative fitness is calculated using the above formula (1).

[0018] However, in the comparative example shown in Figure 3, even if the boundaries of the element activity sections are misaligned, the reliability may be calculated to be high. Specifically, as shown in Figure 4, assume that in the observation information, a time period with a high degree of conformance to the probabilistic model of element activity 1 and a time period with a high degree of conformance to the probabilistic model of element activity 2 occur at an interval. In this case, even if there is a deviation between the correct boundary between element activity section 1 and element activity section 2 and the boundary in the estimation result, each element activity section includes a time period with a high degree of conformance to the probabilistic model of each element activity, so the relative conformance of each element activity section is high. Therefore, when the relative conformance is used as the reliability as in the example of Figure 4, the reliability of the estimation result of the segmentation of element activity sections is evaluated as high, even though it is desired to evaluate the reliability as low.

[0019] In this embodiment, a method is proposed that can deal with partial deviations in the boundaries of elemental action sections. The action section evaluation device according to this embodiment will be described in detail below.

[0020] 5, the action interval evaluation device 10 functionally includes an extraction unit 11, a machine learning unit 20, and a segmentation unit 30. The machine learning unit 20 further includes an observation probability learning unit 21, a transition probability learning unit 22, and a construction unit 23. The segmentation unit 30 further includes an estimation unit 31, a calculation unit 32, and an evaluation unit 33. An estimation model 41 is stored in a predetermined storage area of ​​the action interval evaluation device 10.

[0021] The extraction unit 11 acquires learning video during machine learning. The learning video is video of a person's behavior, and is video to which teacher information indicating action sections indicating time sections of the target behavior and divisions of elemental action sections indicating time sections of each elemental behavior included in the target behavior is added. The extraction unit 11 calculates features related to human movements from the video of the action sections of the learning video and extracts time-series features. The extraction unit 11 also acquires segmentation video during segmentation. The segmentation video is video of a person's behavior, and divisions of elemental action sections in the target behavior section are unknown. Note that in this embodiment, the action sections in the segmentation video are known, for example, because the segmentation video itself is a time section of the action section, or the start and end times of the action section are added to the segmentation video. The extraction unit 11 similarly extracts time-series features from the action sections of the segmentation video.

[0022] An example of a method for extracting time-series features from video by the extraction unit 11 will be described in detail. The extraction unit 11 applies person detection technology to detect human regions (e.g., bounding boxes) from each frame constituting the video (learning video or segmentation video), and tracks the detected regions of the same person by associating them between frames. When multiple human regions are detected from a single frame, the extraction unit 11 identifies the region of the person to be determined based on the size of the region, the position of the region within the frame, etc. The extraction unit 11 performs image processing on the image within the human region detected from each frame, and calculates posture information based on the joint positions of the person and their connection relationships, etc. The extraction unit 11 creates time-series posture information by associating the posture information calculated for each frame with the time information associated with the frame and arranging them.

[0023] The extraction unit 11 also calculates time-series motion information for each body part from the time-series posture information. The motion information may be, for example, the degree of bending of each part, the bending speed, etc. Each part may be, for example, an elbow, a knee, etc. The extraction unit 11 also calculates a feature vector whose elements are values ​​obtained by averaging the motion information within the window in the time direction at regular time intervals using a sliding time window.

[0024] During machine learning, the extraction unit 11 passes the extracted time series features and teaching information indicating the divisions of the action sections and element action sections contained in the learning video to the machine learning unit 20 as supervised data, and during segmentation, passes the extracted time series features to the segmentation unit 30.

[0025] The machine learning unit 20 generates an estimation model 41 by performing machine learning using the supervised data passed from the extraction unit 11 .

[0026] In this embodiment, a hidden semi-Markov model (hereinafter referred to as "HSMM (Hidden semi-Markov model)") as shown in Fig. 6 is constructed as an example of an estimation model 41 that estimates the segmentation of element action intervals in a target action interval. In addition to the parameters of a hidden Markov model (hereinafter referred to as "HMM (Hidden Markov model)"), the HSMM has a probability distribution of the duration of each state as a parameter.

[0027] The HSMM of this embodiment includes a plurality of first HMMs that represent each of human actions as a state, and a second HMM that represents elemental actions as states. In FIG. 6, m1, m2, and m3 are states corresponding to each action, and a1, a2, and a3 are states corresponding to each elemental action. An elemental action is a combination of multiple actions, and an action is a combination of multiple postures. When a time-series feature quantity related to human actions extracted from video is given to the HSMM constructed by setting parameters, the HSMM estimates optimal segmentation of elemental action sections. In FIG. 6, d1, d2, and d3 are examples of elemental action sections.

[0028] HMM parameters include observation probabilities and transition probabilities. In FIG. 6, O1, O2, ..., O8 are examples of observation probabilities, and transition probabilities are associated with arrows connecting states. An observation probability is the probability that certain observation information is observed in each state, and a transition probability is the probability of transitioning from one state to another. If the order of transitions is fixed, transition probabilities are unnecessary. Note that the number of actions and the number of element actions, i.e., the number of first HMMs and second HMMs, are merely examples and are not limited to the numbers exemplified in FIG. 6. The observation probability learning unit 21, transition probability learning unit 22, and construction unit 23 included in the machine learning unit 20 will be described in detail below.

[0029] As described below, the observation probability learning unit 21 learns the observation probability of each action constituting the HSMM, which is an example of the estimation model 41, using time series features (hereinafter also referred to as "unsupervised data") obtained by removing the supervised information from the supervised data.

[0030] In this embodiment, the target behavior is a limited behavior for achieving a certain work goal. Such behavior is, for example, a behavior in a routine task performed on a factory line, and has the following characteristics. Property 1: The differences between the component actions that make up a behavior are differences in the combination of a limited number of actions. Property 2: Multiple postures observed when performing the same action are similar.

[0031] In this embodiment, based on property 1, all elemental actions are composed of actions included in one action group. For example, as shown in FIG. 7, the action group includes three actions m11, m12, and m13. For example, action m11 may be "raising the arm", action m12 may be "lowering the arm", and action m13 may be "stretching the arm forward". The number of actions included in the action group is not limited to the example in FIG. 7. Furthermore, the number of actions included in each elemental action is not limited to the example in FIG. 7.

[0032] For example, the observation probability learning unit 21 calculates the observation probability of each action using a Gaussian mixture model (hereinafter referred to as "GMM (Gaussian Mixture Model)"). Specifically, the observation probability learning unit 21 estimates parameters of a GMM in which Gaussian distributions corresponding to the number of actions are mixed, by clustering the feature amounts passed from the extraction unit 11. Then, the observation probability learning unit 21 assigns each Gaussian distribution constituting the GMM whose parameters have been estimated as a probability distribution representing the observation probability of each action.

[0033] As will be described below, the transition probability learning unit 22 calculates the transition probabilities between actions represented by the first HMM based on supervised data. Specifically, the transition probability learning unit 22 allocates time-series feature quantities to each element action interval based on the supervised information contained in the supervised data. Then, the transition probability learning unit 22 uses the time-series feature quantities allocated to each element action interval as observation information, fixes the observation probability of each action calculated by the observation probability learning unit 21, and calculates the transition probabilities between actions using, for example, maximum likelihood estimation or an EM algorithm (Expectation-Maximization algorithm).

[0034] It should be noted that since creating supervised data takes time and effort, the transition probability learning unit 22 may pad the supervised data by adding noise to the seed supervised data, for example.

[0035] The construction unit 23 sets a probability distribution of the duration of each element action based on the duration of each element action interval given in the teacher information. For example, the construction unit 23 sets a uniform distribution within a predetermined range for the duration of each element action interval given in the teacher information as the probability distribution of the duration of the element action.

[0036] The construction unit 23 constructs an HSMM, such as that shown in FIG. 6, as an estimation model 41 using the observation probability of each action calculated by the observation probability learning unit 21, the transition probability between actions calculated by the transition probability learning unit 22, and the duration of each set element action. The estimation model 41 is an HSMM in which the second HMM corresponding to each element action transitions in the order of the element actions given in the training information after the set duration. In FIG. 6, O1, O2, . . . , O8 represent observation probabilities calculated by the observation probability learning unit 21. Furthermore, the transition probabilities associated with the arrows between actions m1, m2, and m3 included in element actions a1, a2, and a3, respectively, correspond to the transition probabilities calculated by the transition probability learning unit 22. Furthermore, d1, d2, and d3 represent the duration of each element action. The construction unit 23 stores the constructed estimation model 41 in a predetermined storage area.

[0037] The segmentation unit 30 estimates the segmentation of elemental action sections in the target action section of the segmentation video based on the time-series feature quantities passed from the extraction unit 11, and evaluates the estimation results of the segmentation of elemental action sections. The estimation unit 31, calculation unit 32, and evaluation unit 33 included in the segmentation unit 30 will be described in detail below.

[0038] The estimation unit 31 estimates the segmentation of elemental action segments in an action segment by inputting time-series feature quantities corresponding to the action segments of the segmentation video to the estimation model 41. The estimation unit 31 passes the estimation result of the segmentation of elemental action segments to the calculation unit 32.

[0039] The calculation unit 32 calculates the fitness of the time series features corresponding to each of the element action intervals before and after the boundary when the boundary between adjacent element action intervals is changed, to the second HMM, which is a probabilistic model corresponding to the element action in question.

[0040] Specifically, the calculation unit 32 calculates the degree of conformance as follows, using the boundary j of the element action in the estimation result of the segmentation of the element action section as a reference, while shifting the time corresponding to the boundary j. The calculation unit 32 calculates the degree of conformance as follows, using the time series feature quantity {X j ,Xj+1}, the fitness P(X j ,X j+1 |M j ,M j+1 ) is calculated. j ,X j+1 |M j ,M j+1 )=P(X j |M j )P(X j+1 |M j+1 ) and M j and M j+1 is a probabilistic model (second HMM) of element actions before and after boundary j. The calculation unit 32 calculates P(X j ,X j+1 |M j ,M j+1 ) as a function E of time t corresponding to boundary j j It is expressed as (t).

[0041] For example, as shown in Figure 8, the feature quantity at time t is x t (In the example of FIG. 8, t=1, 2, . . . , 8). Also, the time T j If t, then x t The feature values ​​up to j are taken as the observation information of the j-th elemental action. j = 4, the calculation unit 32 calculates the features x1 to x4 as a probability model M j Enter P(X j |M j ) and the features x5 to x8 are used to create the probabilistic model M j+1 Enter P(X j+1 |M j+1 ) and the calculation unit 32 calculates P(X j |M j ) and P(X j+1 |M j+1 ) to E j (T j Similarly, the calculation unit 32 calculates E j The middle diagram on the left in Figure 8 shows the boundary j at time T j Time T one time before j This is an example where the fitness is set to -1, and the fitness in this case is E j (T j-1). In addition, the bottom diagram on the left side of Figure 8 shows the boundary j at time T j Time T after one time j +1, the fitness in this case is E j (T j +1). The calculation unit 32 calculates the function E j (t) is passed to the evaluation unit 33.

[0042] Here, the time T j DES j The segmentation of the elemental action interval should be estimated so that (t) is maximized. Therefore, as shown in the right diagram of Figure 8, when the time corresponding to the boundary j is shifted, E j (t) decreases, and the larger the time interval, the greater the E j When the time corresponding to boundary j is clear from the feature, that is, when there are almost no other boundaries, the time corresponding to boundary j is set as T j When changing from E j On the other hand, when the boundary is unclear, the time corresponding to the boundary j is set as T j Even if you change it from E j (t) does not drop significantly immediately and remains at a relatively high value. j If the value of (t) remains high, it can be said that the reliability of the estimated results of the segmentation of the element action section is low.

[0043] Therefore, the evaluation unit 33 evaluates the reliability of the estimation result of the segmentation of the element action interval based on the change in the degree of conformance according to the degree of change in the boundary. Specifically, the evaluation unit 33 evaluates the reliability of the estimation result of the segmentation of the element action interval based on the change in the degree of conformance according to the degree of change in the boundary. j When the boundary is changed based on the above, the evaluation value calculated decreases as the maximum value of the time width in which the change in the degree of conformance falls within the predetermined range increases.

[0044] More specifically, the evaluation unit 33 evaluates the time T j The logarithm of the fitness when log(E j(T j )) as a reference, the evaluation unit 33 calculates the maximum time for which the logarithm of the goodness of fit falls within a certain range when the boundary j is changed. For example, as shown in FIG. 11, the evaluation unit 33 calculates the maximum time for which the logarithm of the goodness of fit falls within a certain range when the boundary j is changed. j (t))>log(E j (T j ))-C (C is a constant determined experimentally) in the range where ||tT j The maximum value of || is Δt j , t at that time is T j 'Let's say.

[0045] In addition, the evaluation unit 33 j For example, the evaluation unit 33 calculates the reliability of the element activity section j when focusing on the boundary j so that the reliability decreases as the length L of the estimated time section of the element activity section j increases. j , L j =T j -T j Then, the evaluation unit 33 determines whether the time corresponding to the boundary j is T j From T j If the length of the element action interval j becomes longer when the element action interval is changed to ', that is, T j ≦T j In the case of ', the confidence level is j / (L j +Δt j On the other hand, the evaluation unit 33 calculates the time corresponding to the boundary j as T j From T j If the length of the element action interval j is shortened when the element action interval is changed to ', that is, T j >T j ', the confidence level is (L j -Δt j ) / L j It is calculated as follows.

[0046] Focusing on one element action section, reliability is calculated for each of the two boundaries, the start and end of the section. The evaluation unit 33 integrates the two reliability values ​​to calculate the final reliability of the element action section. The two reliability values ​​may be integrated by selecting the minimum value of the two reliability values ​​or by averaging the two reliability values. Note that for the first element action section of an action section, since there is no element action section before it, the evaluation unit 33 adopts the reliability calculated for the boundary at the end of the section. Furthermore, for the last element action section of an action section, since there is no element action section after it, the evaluation unit 33 adopts the reliability calculated for the boundary at the start of the section. The evaluation unit 33 assigns the calculated reliability for each element action section to the estimation result of the segmentation of the element action section, and outputs it as an estimated result with an evaluation.

[0047] The action interval evaluation device 10 may be realized by, for example, a computer 50 shown in Fig. 12. The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input / output device 54 such as an input unit and a display unit, and an R / W (Read / Write) unit 55 that controls reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I / F (Interface) 56 that is connected to a network such as the Internet. The CPU 51, memory 52, storage unit 53, input / output device 54, R / W unit 55, and communication I / F 56 are connected to one another via a bus 57.

[0048] The storage unit 53 may be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 53 serving as a storage medium stores an action interval evaluation program 60 for causing the computer 50 to function as the action interval evaluation device 10. The action interval evaluation program 60 includes an extraction process 61, a machine learning process 70, and a segmentation process 80. The storage unit 53 also includes an information storage area 90 in which information constituting the estimation model 41 is stored.

[0049] The CPU 51 reads out the action interval evaluation program 60 from the storage unit 53, loads it in the memory 52, and sequentially executes the processes of the action interval evaluation program 60. The CPU 51 operates as the extraction unit 11 shown in FIG. 5 by executing the extraction process 61. The CPU 51 also operates as the machine learning unit 20 shown in FIG. 5 by executing the machine learning process 70. The CPU 51 also operates as the segmentation unit 30 shown in FIG. 5 by executing the segmentation process 80. The CPU 51 also reads out information from the information storage area 90 and loads the estimation model 41 in the memory 52. ​​As a result, the computer 50 that has executed the action interval evaluation program 60 functions as the action interval evaluation device 10. The CPU 51 that executes the program is hardware.

[0050] The functions realized by the action interval evaluation program 60 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit).

[0051] Next, the operation of the action section evaluation device 10 according to this embodiment will be described. When a learning video is input to the action section evaluation device 10 and an instruction is given to execute machine learning of the estimation model 41, the action section evaluation device 10 executes the machine learning process shown in Fig. 13. Furthermore, when a segmentation video is input to the action section evaluation device 10 and an instruction is given to execute segmentation of the target action section, the action section evaluation device 10 executes the segmentation process shown in Fig. 14. Note that the machine learning process and the segmentation process are examples of the action section evaluation method of the disclosed technology.

[0052] First, the machine learning process shown in FIG. 13 will be described.

[0053] In step S11, the extraction unit 11 acquires the learning video input to the activity section evaluation device 10, and extracts time-series feature quantities related to human movements from the video of the activity section of the learning video.

[0054] Next, in step S12, the observation probability learning unit 21 estimates the parameters of a GMM that contains a mixture of Gaussian distributions equal to the number of actions by clustering the features extracted in step S11. Then, the observation probability learning unit 21 assigns each Gaussian distribution that constitutes the GMM whose parameters have been estimated as a probability distribution that represents the observation probability of each action.

[0055] Next, in step S13, the transition probability learning unit 22 allocates the time-series feature quantities extracted in step S11 to each element action interval indicated by the supervised information in the supervised data. Next, in step S14, the transition probability learning unit 22 uses the time-series feature quantities allocated to each element action interval as observed data, fixes the observation probability of each action calculated in step S12, and calculates the transition probability between actions.

[0056] Next, in step S15, the construction unit 23 sets a probability distribution of duration for each element action based on the duration of each element action interval provided in the training information. Next, in step S16, the construction unit 23 constructs an HSMM as an estimation model 41 using the observation probability of each action calculated in step S12, the transition probability between actions calculated in step S14, and the duration of each element action set in step S15. Then, the construction unit 23 stores the constructed estimation model 41 in a predetermined storage area, and the machine learning process ends.

[0057] Next, the segmentation process shown in FIG. 14 will be described.

[0058] In step S21, the extraction unit 11 acquires the segmentation video input to the action segment evaluation device 10 and extracts time-series feature amounts related to human movements from the action segments of the segmentation video. Next, in step S22, the estimation unit 31 inputs the time-series feature amounts corresponding to the action segments of the segmentation video to the estimation model 41, thereby estimating the segmentation of element action segments in the action segment.

[0059] Next, in step S23, the calculation unit 32 extracts a boundary j for each element action section j included in the action section. Then, the calculation unit 32 calculates the feature quantity {X j ,X j+1}, the probability model M j and M j+1 The fitness for the function E when the boundary j is shifted at each time j Calculated as (t).

[0060] Next, in step S24, the evaluation unit 33 evaluates the time T j The logarithm of the fitness when log(E j (T j )) based on the maximum time Δt when the logarithm of the fitness falls within a specified range when the boundary j is changed. j Next, in step S25, the evaluation unit 33 calculates the maximum time Δt j The reliability of the element activity section j is calculated so that the reliability decreases as σ increases relative to the length of the element activity section j. The above steps S23 to S25 are executed for all boundaries included in the activity section.

[0061] Next, in step S26, the evaluation unit 33 integrates the two reliability scores calculated for the two boundaries, the start and end of the element activity interval, to calculate the final reliability score for each element activity interval. Next, in step S27, the evaluation unit 33 assigns the calculated reliability score for each element activity interval to the estimation result of the segmentation of the element activity interval, outputs it as an estimated result with an evaluation, and the segmentation process ends.

[0062] As described above, the activity section evaluation device according to this embodiment estimates the segmentation of element activity sections based on time-series feature values ​​extracted from video of a person's activity and an estimation model including a probabilistic model for each of the multiple element activity sections included in the activity. Furthermore, when a boundary between adjacent element activity sections is changed, the activity section evaluation device calculates the degree of conformance of the time-series feature values ​​corresponding to each element activity section before and after the boundary to the probabilistic model corresponding to the element activity. The activity section evaluation device then evaluates the estimation result of the segmentation of element activity sections based on the change in conformance depending on the degree of boundary change. This makes it possible to accurately evaluate the reliability of the segmentation of a specified activity in a video of a person. In other words, the activity section evaluation device according to this embodiment improves the functionality of a computer.

[0063] In the above embodiment, the estimation model is described as an HSMM, but the present invention is not limited to this, and other machine learning models such as a model using a neural network may also be applied.

[0064] In addition, in the above embodiment, in addition to the evaluation value based on the change in the degree of conformance according to the degree of boundary change, the final reliability may also be calculated by taking into account the reliability calculated from the relative conformance of the comparative example described in Figure 3 and taking the minimum value of these.

[0065] In the above embodiment, the action section evaluation device estimates the segmentation of element action sections in the action section of the segmentation video, but the estimation result of the segmentation of element action sections may be acquired instead of the segmentation video. In this case, the segmentation unit 30 may include an acquisition unit that acquires the estimation result of the segmentation of element action sections, and the estimation unit 31 may be omitted.

[0066] Furthermore, in the above embodiment, the case where the action section evaluation device including the machine learning unit and the segmentation unit is realized by one computer has been described, but each may be realized by a separate computer.

[0067] In the above embodiment, the behavior section evaluation program is pre-stored (installed) in the storage unit, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored on a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory. [Explanation of symbols]

[0068] 10. Action interval evaluation device 11 Extraction part 20 Machine Learning Department 21 Observation Probability Learning Unit 22 Transition probability learning unit 23 Construction Department 30 Segment 31 Estimation part 32 Calculation section 33 Evaluation Department 41 Estimation Model 50 Computers 51 CPU 52 memory 53 Storage section 54 Input / Output Devices 55 R / W section 56 Communication I / F 57 Bus 59 Storage medium 60 Action Section Evaluation Program 61 Extraction Process 70 Machine Learning Process 80 Segmentation Process 90 Information storage area

Claims

1. obtaining an estimation result of segmentation of an element behavior interval, which is a time interval of the element behavior, based on a time series feature amount extracted from a video of a human behavior and an estimation model including a probabilistic model for each of a plurality of element behaviors included in the behavior; calculating a degree of conformity of the feature quantity of the time series corresponding to each of the element action sections before and after the boundary, when a boundary between adjacent element action sections is changed, to the probabilistic model corresponding to the element action; Evaluating the estimation result of the segmentation of the element action section based on the change in the degree of conformance according to the degree of change in the boundary. An action interval evaluation program for causing a computer to execute a process including the above.

2. 2. The action interval evaluation program according to claim 1, wherein the process of evaluating the estimation result includes evaluating the reliability of the estimation result as being lower the greater the maximum value of the time width within which the change in the degree of conformance falls within a predetermined range when the boundary is changed based on the time that is the boundary indicated by the estimation result.

3. 3. The activity interval evaluation program according to claim 2, wherein the process of evaluating the estimation result includes evaluating the estimation result to have a lower reliability as the maximum value of the time width relative to the length of the element activity interval increases.

4. 4. The activity section evaluation program according to claim 3, wherein the process of evaluating the estimation result includes calculating a reliability based on a maximum value of the time width with respect to the length of the element activity section, and integrating the reliability calculated when a boundary corresponding to a start time of the element activity section is changed with the reliability calculated when a boundary corresponding to an end time of the element activity section is changed to calculate a final reliability.

5. Extracting the time series of features from a video of the person's behavior; A segmentation of the element action section is estimated based on the extracted time series feature amount and the estimation model.

5. The action interval evaluation program according to claim 1, further comprising:

6. 6. The action section evaluation program according to claim 1, further comprising: generating the estimation model by performing machine learning using as training information a time-series feature quantity extracted from a video of an action section in which segmentation of elemental action sections is known.

7. the action being an action of a person manufacturing a product; The elemental actions are combinations of the actions of the person when performing each step of the manufacturing process. The action interval evaluation program according to any one of claims 1 to 6.

8. an acquisition unit that acquires an estimation result of a segmentation of an element behavior section, which is a time section of the element behavior, based on a time series feature amount extracted from a video of a human behavior and an estimation model including a probabilistic model for each of a plurality of element behaviors included in the behavior; a calculation unit that calculates a degree of conformity of the feature quantity of the time series corresponding to each of the element action sections before and after a boundary between adjacent element action sections when the boundary is changed, to the probabilistic model corresponding to the element action; an evaluation unit that evaluates an estimation result of the segmentation of the element action section based on a change in the degree of conformance according to a degree of change in the boundary; An action interval evaluation device comprising:

9. obtaining an estimation result of segmentation of an element behavior interval, which is a time interval of the element behavior, based on a time series feature amount extracted from a video of a human behavior and an estimation model including a probabilistic model for each of a plurality of element behaviors included in the behavior; calculating a degree of conformity of the feature quantity of the time series corresponding to each of the element action sections before and after the boundary, when a boundary between adjacent element action sections is changed, to the probabilistic model corresponding to the element action; Evaluating the estimation result of the segmentation of the element action section based on the change in the degree of conformance according to the degree of change in the boundary. The action interval evaluation method includes a process performed by a computer, the process including:

Images