Periodic motion detection device, periodic motion detection method, and periodic motion detection program

Abstract

To provide a periodic movement detection device that is able to detect periodic movements including a plurality of movements of different lengths.SOLUTION: A periodic movement detection device includes: a clustering unit 13 for clustering feature amounts of time-series motion extracted from a moving image 1Ba related to work of a worker; a series pattern mining unit 16 for identifying a frequent pattern from time-series data of a cluster that has been classified; a frequent pattern extraction unit 17 for creating two frequent pattern sets by extracting the identified frequent pattern in two different methods when the frequent pattern is extracted from the time-series data; and a periodic movement detection unit 18 for detecting a periodic movement on the basis of either one of the two sets.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a periodic action detection device, a periodic action detection method, and a periodic action detection program. [Background technology]

[0002] The following Patent Document 1 discloses a periodic motion detection method for detecting periodic motions, such as hand movements, from moving images. This detection method performs frequency transformation (Fourier transformation) on a predetermined region of each of a plurality of frame images, and determines whether or not the periodic motion to be detected exists in the plurality of frame images based on the frequency of the transformation result and the frequency of the periodic motion to be detected. This detection method requires that the time lengths of each motion be the same, since comparison is made using frequency transformation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-123538 A Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, the periodic motions of a worker working at a production site such as a factory include a plurality of motions of different lengths, such as the motion of grasping a part stored in a parts box, the motion of transporting the grasped part to a work space, the motion of adjusting the transported part by assembling it, the motion of storing the finished product in a storage location, and the like.

[0005] When trying to detect such periodic movements that include multiple movements from a moving image, the detection method of Patent Document 1, which requires that the lengths of the movements be uniform, is unable to detect each movement.

[0006] Therefore, the present invention provides a periodic motion detection device, a periodic motion detection method, and a periodic motion detection program capable of detecting a periodic motion including a plurality of motions with different lengths. [Means for solving the problem]

[0007] A periodic motion detection device according to one aspect of the present disclosure includes a clustering unit that classifies time-series movement features extracted from time-series information related to worker tasks by clustering and outputs the classified clusters as one-dimensional time-series data, a labeling unit that executes a labeling process to combine multiple consecutive identical clusters into one cluster from the one-dimensional time-series data that has been subjected to the labeling process, a sequential pattern mining unit that identifies a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time-series data that has been subjected to the labeling process, and a sequential pattern mining unit that, when extracting the identified frequent pattern from the one-dimensional time-series data, configures a series of infrequent patterns located immediately after the frequent pattern to be extracted. The apparatus includes a first frequent pattern extraction unit that concatenates a series of clusters to the end of a frequent pattern to be extracted and sequentially extracts it as a single frequent pattern; a second frequent pattern extraction unit that, when extracting an identified frequent pattern from one-dimensional time series data, concatenates a series of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted to the beginning of the frequent pattern to be extracted and sequentially extracts it as a single frequent pattern; and a periodic behavior detection unit that detects periodic behavior based on either the set of frequent patterns extracted by the first frequent pattern extraction unit or the set of frequent patterns extracted by the second frequent pattern extraction unit.

[0008] According to this aspect, it is possible to classify the time-series movement feature values ​​extracted from the time-series information related to the work of the worker by clustering, identify a series of clusters that are frequent patterns from the time-series data of the classified clusters, and create two sets by using two different methods to extract the identified frequent pattern together with infrequent patterns existing before and after it, and to detect the periodic movement of the worker based on one of the sets. That is, it is possible to cluster the time-series movement feature values ​​extracted from the time-series information and arrange them in a chronological order, and to identify the periodic movement of the worker based on the series of frequent clusters identified from the series of clusters arranged in that time order.

[0009] In the above aspect, the device may further include an output unit that outputs the number of occurrences of the periodic operation, a start point of the periodic operation, and an end point of the periodic operation based on the periodic operation detected by the periodic operation detection unit.

[0010] This makes it possible to easily recognize multiple movements of different lengths included in the worker's periodic movements based on the start and end points of the periodic movements and the number of times the periodic movements occur.

[0011] In the above aspect, the periodic behavior detection unit may compare an average value of similarity between frequent patterns included in the set of frequent patterns extracted by the first frequent pattern extraction unit with an average value of similarity between frequent patterns included in the set of frequent patterns extracted by the second frequent pattern extraction unit, and detect periodic behavior based on the set with the higher average similarity.

[0012] This makes it possible to detect the periodic movements of a worker using the set that has the higher degree of similarity between the elements that make up the set, thereby improving the detection accuracy when detecting periodic movements.

[0013] In the above aspect, the device may further include an evaluation unit that evaluates the positional shift of the detected periodic behavior based on the degree of overlap between an area of ​​a detected periodic behavior in which the periodic behaviors detected by the periodic behavior detection unit are arranged in chronological order, and an area of ​​an exemplary periodic behavior in which exemplary periodic behaviors are arranged in chronological order, and the output unit may further output the content evaluated by the evaluation unit.

[0014] This makes it possible to provide an objective evaluation result calculated based on the positional deviation from a model periodic motion.

[0015] In the above aspect, the system may further include an exclusion unit that excludes a first cluster and a last cluster as excluded clusters from the one-dimensional time series data that has been subjected to the labeling process, and the sequential pattern mining unit may identify a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time series data from which the excluded clusters have been excluded.

[0016] This makes it possible to exclude actions such as preparation and cleanup that are performed before or after the start of work from the processing targets, thereby improving the processing efficiency when detecting periodic actions.

[0017] A periodic motion detection method according to another aspect of the present disclosure is a method executed by a processor, the method including a clustering step of classifying time-series movement features extracted from time-series information on a worker's work by clustering and outputting the classified clusters as one-dimensional time-series data, a labeling step of performing a labeling process to group multiple consecutive identical clusters into one cluster from the one-dimensional time-series data that has been subjected to the labeling process, a sequential pattern mining step of identifying a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time-series data that has been subjected to the labeling process, and a sequence of infrequent patterns located immediately after the frequent pattern to be extracted when extracting the identified frequent pattern from the one-dimensional time-series data. The method includes a first frequent pattern extraction step of sequentially extracting a single frequent pattern by concatenating a sequence of clusters constituting a turn to the end of a frequent pattern to be extracted; a second frequent pattern extraction step of sequentially extracting a single frequent pattern by concatenating a sequence of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted to the beginning of the frequent pattern to be extracted when extracting a specified frequent pattern from one-dimensional time series data; and a periodic action detection step of detecting a periodic action based on either the set of frequent patterns extracted in the first frequent pattern extraction step or the set of frequent patterns extracted in the second frequent pattern extraction step.

[0018] According to this aspect, the feature values ​​of the time-series movements extracted from the time-series information on the work of the worker are classified by clustering, a series of clusters that are frequent patterns are identified from the time-series data of the classified clusters, and the identified frequent patterns are extracted together with infrequent patterns existing before and after the frequent patterns by using two different methods to create two sets, and periodic movements of the worker can be detected based on one of the sets. That is, the feature values ​​of the time-series movements extracted from the time-series information are clustered and arranged in a time series, and periodic movements of the worker can be identified based on the series of frequent clusters identified from the series of clusters arranged in that time series.

[0019] A periodic motion detection program according to another aspect of the present disclosure includes a computer, the computer comprising: a clustering unit that classifies, by clustering, feature quantities of time-series movements extracted from time-series information on tasks of a worker and outputs the classified clusters as one-dimensional time-series data; a labeling unit that executes a labeling process to combine multiple consecutive identical clusters into one cluster from the one-dimensional time-series data that has been subjected to the labeling process; a sequential pattern mining unit that identifies a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time-series data that has been subjected to the labeling process; and a sequence pattern mining unit that, when extracting the identified frequent pattern from the one-dimensional time-series data, identifies a series of infrequent patterns located immediately after the frequent pattern to be extracted. a first frequent pattern extraction unit that sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately before a frequent pattern to be extracted as a single frequent pattern by linking the series of clusters constituting the series of infrequent patterns located immediately before the frequent pattern to be extracted as a single frequent pattern when extracting a specified frequent pattern from one-dimensional time-series data, to the beginning of the frequent pattern to be extracted; and a periodic behavior detection unit that detects a periodic behavior based on either the set of frequent patterns extracted by the first frequent pattern extraction unit or the set of frequent patterns extracted by the second frequent pattern extraction unit.

[0020] According to this aspect, the feature values ​​of the time-series movements extracted from the time-series information on the work of the worker are classified by clustering, a series of clusters that are frequent patterns are identified from the time-series data of the classified clusters, and the identified frequent patterns are extracted together with infrequent patterns existing before and after the frequent patterns by using two different methods to create two sets, and periodic movements of the worker can be detected based on one of the sets. That is, the feature values ​​of the time-series movements extracted from the time-series information are clustered and arranged in a time series, and periodic movements of the worker can be identified based on the series of frequent clusters identified from the series of clusters arranged in that time series. Effect of the Invention

[0021] According to the present invention, it is possible to provide a periodic movement detection device, a periodic movement detection method, and a periodic movement detection program capable of detecting a periodic movement including a plurality of movements of different lengths. [Brief description of the drawings]

[0022] [Figure 1] 1 is a diagram illustrating an example of an overview of a system including a periodic operation detection device according to an embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating an example of a functional configuration of a system including a periodic operation detection device. [Diagram 3] 1 is a schematic diagram conceptually illustrating a flow of a periodic motion detection process in a periodic motion detection device; [Figure 4] 11A and 11B are schematic diagrams illustrating data before and after a clustering process is performed. [Diagram 5] 11A and 11B are schematic diagrams illustrating data transition states in a periodic operation detection process; [Figure 6] 13 is an example of a graph in which the detection results of periodic actions are output in a time series. [Figure 7] FIG. 2 is a diagram illustrating an example of a hardware configuration of a periodic operation detection device. [Figure 8] 4 is a flowchart illustrating an example of an operation of the periodic operation detection device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, referred to as "the present embodiment") will be described with reference to the drawings. In each drawing, the same reference numerals denote the same or similar configurations.

[0024] §1 Examples of application First, an example of a situation to which the present invention is applied will be described with reference to Fig. 1. In a system 100 including a periodic motion detection device 10 according to this embodiment, the image sensors 20a, 20b, and 20c capture the motion of a worker A in a certain work area R, and the periodic motion detection device 10 acquires the captured video, classifies the time-series motion feature values ​​extracted from the video by clustering, identifies a series of clusters that are frequent patterns from the time-series data of the classified clusters, creates two sets by extracting the identified frequent patterns using two different methods, and detects periodic motion based on one of the sets.

[0025] There are two methods for extracting frequent patterns, as follows: In these two methods, a sequence of one or more clusters between adjacent frequent patterns is treated as a non-frequent pattern.

[0026] The first extraction method is a method in which a frequent pattern and a non-frequent pattern located immediately after that frequent pattern are sequentially extracted as one frequent pattern, and a first set is created from the extracted frequent patterns.

[0027] The second extraction method is a method in which a frequent pattern and a non-frequent pattern located immediately before that frequent pattern are sequentially extracted as one frequent pattern, and a second set is created from the extracted frequent patterns.

[0028] Then, the average similarity between the frequent patterns included in the first set is compared with the average similarity between the frequent patterns included in the second set, and periodic behavior is detected based on the set with the higher average similarity.

[0029] Furthermore, based on the detected periodic motion, the number of occurrences of the periodic motion, the start point of the periodic motion, and the end point of the periodic motion are identified.

[0030] This makes it possible to cluster the movement features extracted from the video and arrange them in time series, and to identify periodic movements based on a frequent cluster sequence identified from the sequence of clusters arranged in time series. Therefore, it becomes possible to detect periodic movements that include multiple movements of different lengths.

[0031] That is, the periodic movement detection device 10 according to this embodiment can detect a periodic movement including a plurality of movements having different lengths.

[0032] §2 Configuration example [Function configuration] Next, referring to FIG. 2, an example of the functional configuration of a system 100 including a periodic motion detection device 10 according to the present embodiment will be described. The system 100 includes three image sensors 20a, 20b, and 20c, and a periodic motion detection device 10. In the following, when it is not necessary to distinguish between the three image sensors 20a, 20b, and 20c, they will be referred to as an image sensor 20. The periodic motion detection device 10 has, as its functional configuration, for example, an acquisition unit 11, an extraction unit 12, a clustering unit 13, a labeling unit 14, an exclusion unit 15, a sequence pattern mining unit 16, a frequent pattern extraction unit 17, a periodic motion detection unit 18, an evaluation unit 19, an output unit 1A, and a storage unit 1B. The storage unit 1B stores, for example, a video 1Ba and a trained model 1Bb.

[0033] The functional components of the system 100 including the periodic motion detection device 10 will be described in detail below.

[0034] <Image sensor> The image sensor 20 is, for example, a general-purpose camera, and captures a video including a scene in which the worker A is performing an action in the working area R. The image sensor 20 has, for example, a detection unit as a functional configuration. The detection unit detects the action of the worker A, and outputs a video showing the action as time-series information.

[0035] Each of the image sensors 20a, 20b, 20c is disposed so as to capture the entire working area R and the entire body of the worker A. In this case, for example, each of the image sensors 20a, 20b, 20c may be disposed so as to capture the entire working area R and the entire body of the worker A, or each of the image sensors 20a, 20b, 20c may be disposed so as to capture a part of the working area R and the worker A and to cover the entire working area R and the entire body of the worker A by combining the respective videos. Also, each of the image sensors 20a, 20b, 20c may capture the working area R and the worker A at different magnifications. It is not necessary to provide three image sensors 20, and it is sufficient to provide at least one or more image sensors.

[0036] <Acquisition part> The acquisition unit 11 acquires time-series information (video in this embodiment) related to the actions performed by the worker A from the image sensor 20. The time-series information acquired by the acquisition unit 11 is transmitted to the storage unit 1B and stored as a video 1Ba. The acquisition unit 11 also acquires the video 1Ba stored in the storage unit 1B.

[0037] <Extraction part> The extraction unit 12 executes a movement characteristic extraction process for extracting a characteristic amount of the worker's movement from the images of the moving image 1Ba.

[0038] A specific description will be given with reference to Fig. 3. As shown in the upper part of Fig. 3, the extraction unit 12 detects skeletal data Da indicating the movement of the worker's skeleton from the image of the video 1Ba, and extracts a feature amount Db of the movement based on the detected skeletal data Da. As a method for detecting the skeletal data, for example, a known skeletal detection method such as OpenPose can be used.

[0039] Furthermore, when extracting the movement feature amount Db based on the skeleton data Da, the learned model 1Bb may be used. As the learned model 1Bb, for example, a known model such as an autoencoder can be used. Specifically, the extraction unit 12 inputs the skeleton data Da to an autoencoder, and extracts only the movement component data from the data (for example, movement component data, skeleton component data, and viewpoint component data) encoded and separated by the autoencoder, thereby extracting the movement feature amount Db.

[0040] Here, the movement feature extraction process is not limited to the above process. For example, it may be a process of extracting the movement feature amount Db of the worker from the image of the video 1Ba. As a method for extracting the movement feature amount from the image, for example, a known feature amount extraction method such as SIFT (Scale-Invariant Feature Transform) or SURF (Speed ​​Up Robust Feature) can be used.

[0041] <Clustering Department> The clustering unit 13 in FIG. 2 executes a clustering process based on the feature amount Db of the time-series movement extracted by the extraction unit 12.

[0042] A specific description will be given with reference to Fig. 3. As shown in the middle part of Fig. 3, the clustering unit 13 classifies the time-series movement feature amount Db extracted by the extraction unit 12 by clustering, and outputs the classified clusters as one-dimensional time-series data Dc.

[0043] As an algorithm for clustering, for example, Spectral Clustering can be used. In Spectral Clustering, the number of clusters to be grouped can be set arbitrarily. For example, if the number of clusters is set to 8, 8 groups are created, and each cluster is assigned a cluster number corresponding to the group to which it belongs.

[0044] A specific description will be given with reference to Fig. 4. The figure shows an example of one-dimensional time series data Dc output as a result of executing clustering processing on feature quantities Db of time series movements. Among the clusters constituting this time series data Dc, for example, the first cluster from the left is assigned the cluster number 5, and similarly the second, third, and fourth clusters are assigned the cluster numbers 0, 1, and 2, respectively.

[0045] For example, a group of movements corresponding to grasping can be assigned a cluster number of 0, a group of movements corresponding to carrying can be assigned a cluster number of 1, and a group of movements corresponding to adjusting can be assigned a cluster number of 2. Movements corresponding to grasping can include, for example, movements to grab a box or part, movements corresponding to carrying can include, for example, movements to carry a box or part, and movements corresponding to adjusting can include, for example, movements to fix or install a box or part.

[0046] <Labeling Department> The labeling unit 14 in FIG. 2 executes a labeling process for grouping a plurality of consecutive identical clusters into one cluster from among the one-dimensional time series data output by the clustering unit 13.

[0047] A specific description will be given with reference to Fig. 5. The labeling unit 14 performs labeling processing on one-dimensional time series data Dc shown in Fig. 5(A) and outputs one-dimensional time series data Dd shown in Fig. 5(B). The time series data Dc before the labeling processing includes a portion c1 of two consecutive 0s and a portion c2 of two consecutive 2s. On the other hand, in the time series data Dd after the labeling processing, 0s and 2s are each grouped together (d1 and d2).

[0048] <Exclusion part> The exclusion unit 15 in FIG. 2 executes an exclusion process to exclude the first cluster and the last cluster as excluded clusters from the one-dimensional time series data that has been subjected to the labeling process.

[0049] A specific description will be given with reference to Fig. 5. The exclusion unit 15 sets the first cluster d3 and the last cluster d4 of the clusters constituting the one-dimensional time series data Dd shown in Fig. 5(B) as excluded clusters. The exclusion unit 15 outputs the one-dimensional time series data De shown in Fig. 5(C) after making it possible to exclude the excluded cluster from the subsequent processing targets. The method of excluding the excluded cluster may be to erase the excluded cluster from the time series data Dd, or to make it possible to identify the excluded cluster by adding an exclusion mark to the excluded cluster.

[0050] The reason for excluding the first and last clusters is that before and after the start of a series of tasks, preparations, cleanup, and the like are performed, and operations different from the series of tasks involving periodic operations are performed. By excluding the first and last clusters, in which operations different from the series of tasks are performed, from the time-series data to be processed, it is possible to improve the processing efficiency when detecting periodic operations.

[0051] Here, the clusters to be excluded from the processing are not limited to the first and last clusters. For example, a cluster that is relatively far from other clusters among the clusters constituting one-dimensional time series data may be excluded as an excluded cluster.

[0052] <Sequential Pattern Mining Division> 2 executes sequential pattern mining processing to identify frequently occurring cluster sequences as frequent patterns from the one-dimensional time series data after the exclusion processing. As a method of sequential pattern mining, for example, PrefixSpan can be used.

[0053] A specific description will be given with reference to Fig. 5. The sequential pattern mining unit 16 extracts frequent partial sequential patterns from the sequence of clusters constituting the one-dimensional time series data De shown in Fig. 5(C). The sequential pattern mining unit 16 outputs one-dimensional time series data Df shown in Fig. 5(D) as a result of the sequential pattern mining process. In this time series data Df, a pattern of <0, 1, 2, 3> is identified as a frequent partial sequential pattern.

[0054] <Frequent pattern extraction section> The frequent pattern extraction unit 17 in Fig. 2 executes a frequent pattern extraction process to extract the frequent patterns identified by the sequential pattern mining unit 16 from time series data using two different methods. In these two methods, a sequence of one or more clusters between adjacent frequent patterns is treated as a non-frequent pattern. The two methods will be explained in order below.

[0055] The first extraction method is a method in which a frequent pattern and a non-frequent pattern located immediately after that frequent pattern are sequentially extracted as one frequent pattern, and a first set is created from the extracted frequent patterns.

[0056] A specific description will be given with reference to Fig. 5. When extracting the specified frequent pattern <0,1,2,3> from the one-dimensional time series data Df shown in Fig. 5(D), the frequent pattern extraction unit 17 (first frequent pattern extraction unit) concatenates a series of clusters constituting a series of non-frequent patterns located immediately after the frequent pattern <0,1,2,3> to be extracted to the end of the frequent pattern <0,1,2,3> to be extracted, and sequentially extracts each as one frequent pattern 1a, 1b, 1c as shown in Fig. 5(E). Then, a first set G1 shown in Fig. 5(F) is generated from the extracted frequent patterns 1a, 1b, 1c.

[0057] As shown in FIG. 5(F), the first set G1 includes frequent pattern 1a whose elements are <0, 1, 2, 3, 4>, frequent pattern 1b whose elements are <0, 1, 2, 3, 6, 4>, and frequent pattern 1c whose elements are <0, 1, 2, 3>.

[0058] The second extraction method is a method in which a frequent pattern and a non-frequent pattern located immediately before that frequent pattern are sequentially extracted as one frequent pattern, and a second set is created from the extracted frequent patterns.

[0059] A specific description will be given with reference to Fig. 5. When extracting the specified frequent pattern <0,1,2,3> from the one-dimensional time series data Df shown in Fig. 5(D), the frequent pattern extraction unit 17 (second frequent pattern extraction unit) connects a series of clusters constituting a series of non-frequent patterns located immediately before the frequent pattern <0,1,2,3> to be extracted to the beginning of the frequent pattern <0,1,2,3> to be extracted, and sequentially extracts each as one frequent pattern 2a, 2b, 2c as shown in Fig. 5(E). Then, a second set G2 shown in Fig. 5(F) is generated from the extracted frequent patterns 2a, 2b, 2c.

[0060] As shown in FIG. 5(F), the second set G2 includes a frequent pattern 2a whose elements are <0,1,2,3>, a frequent pattern 2b whose elements are <4,0,1,2,3>, and a frequent pattern 2c whose elements are <6,4,0,1,2,3>.

[0061] <Cyclic motion detection unit> The periodic action detection unit 18 in FIG. 2 executes a periodic action detection process for detecting a periodic action based on one of the two sets generated by the frequent pattern extraction unit 17.

[0062] For example, the periodic behavior detection unit 18 calculates the average value of the similarity between the frequent patterns included in each of the two sets for each set, and detects a periodic behavior based on the set with the higher average value of the similarity.

[0063] As a method for calculating the similarity between frequent patterns, for example, a method for measuring the similarity between time series data such as DTW (Dynamic Time Warping) can be used.

[0064] 5(F), for example, if the average similarity of the first set G1 is 0.87 and the average similarity of the second set G2 is 0.62, the periodic action detection unit 18 detects a periodic action based on the first set G1. In this case, the similarity is expressed as a value between 0 and 1, and the closer to 1 the two are similar to each other, the closer to 1 the similarity becomes.

[0065] By detecting the periodic motion of a worker using the set with the higher average similarity, it is possible to improve the detection accuracy when detecting the periodic motion.

[0066] <Evaluation Department> The evaluation unit 19 in FIG. 2 executes an evaluation process for evaluating the positional deviation of the periodic motion detected by the periodic motion detection unit 18.

[0067] Specifically, the evaluation unit 19 evaluates the positional deviation of the detected periodic behavior based on the degree of overlap between an area of ​​detected periodic behavior in which the detected periodic behavior is arranged in chronological order, and an area of ​​model periodic behavior in which model periodic behavior is arranged in chronological order.

[0068] The position shift can be evaluated using, for example, an evaluation index IoUA calculated by the following formula (1) using IoU (Intersection Over Union). IoU is a known index that indicates the degree to which two regions overlap by dividing the common part of the two regions by the union of the two regions.

[0069]

number

[0070] Here, K in equation (1) is the number of data (number of videos), N is the number of periods contained in one data, and th is the IoU threshold.

[0071] A specific description will be given with reference to a graph showing the detection results of periodic movements in a time series as shown in Fig. 6. The upper part of Fig. 6 displays data showing exemplary periodic movements (hereinafter referred to as "exemplary data"), with Ma, Mb, Mc, Md, Me, and Mf displayed as areas of the exemplary periodic movements. The lower part of Fig. 6 displays data showing periodic movements detected by the periodic movement detection unit 18 (hereinafter referred to as "detected data"), with Da, Db, Dc, Dd, De, and Df displayed as areas of the detected periodic movements. For example, Ma and Da are areas of movements corresponding to grasping, Mb and Db are areas of movements corresponding to carrying, and Mc and Dc are areas of movements corresponding to adjustment.

[0072] In this case, the evaluation unit 19 calculates the degree of overlap between the areas Da, Db, Dc, Dd, De, and Df of the detected periodic operation and the areas Ma, Mb, Mc, Md, Me, and Mf of the exemplary periodic operation using formula (1), and evaluates the position shift of the detected periodic operation based on the calculated evaluation index IoUA. If the IoU indicating the degree of overlap is equal to or greater than the threshold, the value in the brackets [ ] in formula (1) becomes 1. On the other hand, if the IoU indicating the degree of overlap is less than the threshold, the value in the brackets [ ] in formula (1) becomes 0. Also, if only one of the areas of the detected periodic operation and the exemplary periodic operation appears, the value in the brackets [ ] in formula (1) becomes 0.

[0073] <Output section> The output section 1A of FIG. 2 executes output processing based on the periodic operation detected by the periodic operation detection section .

[0074] Specifically, based on the detected periodic motion, the output unit 1A outputs the number of periodic motions, the start point of the periodic motion, the end point of the periodic motion, and the content evaluated by the evaluation unit 19. The number of periodic motions, the start point of the periodic motion, and the end point of the periodic motion can be calculated based on the graph illustrated in FIG.

[0075] This makes it possible to recognize multiple movements of different lengths that are included in the worker's periodic movements and then confirm the evaluation of those movements.

[0076] [Hardware configuration] Next, an example of the hardware configuration of the periodic operation detection device 10 according to this embodiment will be described with reference to FIG. 7. The periodic operation detection device 10 has a CPU (Central Processing Unit) 10a equivalent to a calculation device, a RAM (Random Access Memory) 10b and a ROM (Read Only Memory) 10c equivalent to a storage unit 1B, a communication device 10d, an input device 10e, and a display device 10f. These components are connected to each other via a bus so that they can transmit and receive data to each other. Note that, although the periodic operation detection device 10 is configured by one computer in this embodiment, the periodic operation detection device 10 may be realized by using multiple computers.

[0077] The CPU 10a executes programs stored in the RAM 10b or the ROM 10c and functions as a control unit that performs calculations and processing of data. The CPU 10a receives various input data from the input device 10e or the communication device 10d, and displays the results of calculations performed on the input data on the display device 10f or stores the results in the RAM 10b or the ROM 10c.

[0078] The RAM 10b is made up of, for example, a semiconductor memory device, and stores rewritable data.The ROM 10c is made up of, for example, a semiconductor memory device, and stores readable but non-rewritable data.

[0079] The communication device 10d is an interface that connects the periodic motion detection device 10 to an external device. The communication device 10d is connected to the image sensor 20 via a communication network such as a LAN (Local Area Network) or the Internet, and receives video from the image sensor 20.

[0080] The input device 10e is an interface that accepts data input from a user, and can include, for example, a keyboard, a mouse, and a touch panel.

[0081] The display device 10f is an interface that visually displays the results of calculations performed by the CPU 10a, and can be configured, for example, by an LCD (Liquid Crystal Display).

[0082] The program may be provided by being stored in a computer-readable storage medium such as the RAM 10b or the ROM 10c, or may be provided via a communication network connected by the communication device 10d. In the periodic behavior detection device 10, the CPU 10a executes the program to perform the operations of the acquisition unit 11, the extraction unit 12, the clustering unit 13, the labeling unit 14, the exclusion unit 15, the sequence pattern mining unit 16, the frequent pattern extraction unit 17, the periodic behavior detection unit 18, the evaluation unit 19, and the output unit 1A shown in FIG. 2. Note that these physical configurations are merely examples, and do not necessarily have to be independent configurations. For example, the periodic behavior detection device 10 may include an LSI (Large-Scale Integration) that integrates the CPU 10a with the RAM 10b and the ROM 10c.

[0083] §3 Example of operation FIG. 8 is a flowchart showing an example of the operation of the periodic operation detection device 10 according to this embodiment.

[0084] First, the extraction unit 12 of the periodic motion detection device 10 executes a motion feature extraction process based on an image of the video 1Ba related to the work of a worker, and extracts a feature amount of the worker's motion (step S101).

[0085] Next, the clustering unit 13 of the periodic motion detection device 10 executes a clustering process based on the feature amounts of the worker's movement extracted in the above step S101, and outputs the classified clusters as one-dimensional time series data (step S102).

[0086] Next, the labeling unit 14 of the periodic motion detection device 10 executes a labeling process for grouping a plurality of consecutive identical clusters into one cluster from the one-dimensional time series data output in step S102 (step S103).

[0087] Next, the exclusion unit 15 of the periodic motion detection device 10 executes an exclusion process to exclude the first cluster and the last cluster as excluded clusters from the one-dimensional time series data after the labeling process in the above step S103 (step S104).

[0088] Next, the sequential pattern mining unit 16 of the periodic behavior detection device 10 executes a sequential pattern mining process to identify frequently occurring cluster sequences as frequent patterns from the one-dimensional time series data after the exclusion process in the above step S104 (step S105).

[0089] Next, the frequent pattern extraction unit 17 of the periodic motion detection device 10 executes a frequent pattern extraction process to extract the frequent patterns identified in step S105 above from one-dimensional time series data using two methods, and generates two sets (step S106).

[0090] Next, the periodic motion detection unit 18 of the periodic motion detection device 10 executes a periodic motion detection process for detecting a periodic motion based on one of the two sets generated in step S106 (step S107).

[0091] Next, the evaluation unit 19 of the periodic motion detection device 10 executes an evaluation process for evaluating the positional deviation of the periodic motion detected in step S107 above (step S108).

[0092] Next, the output unit 1A of the periodic motion detection device 10 executes an output process based on the periodic motion detected in the above step S107, and outputs the number of periodic motions, the start point of the periodic motion, the end point of the periodic motion, and the contents evaluated in the above step S108 (step S109). Then, this operation ends.

[0093] As described above, the periodic motion detection device 10 according to this embodiment classifies the feature values ​​Db of time-series motion extracted from a video 1Ba related to a worker's work by clustering, identifies frequent patterns from the time-series data Dc of the classified clusters, extracts the identified frequent patterns by two different methods to create a first set and a second set, and can detect the periodic motion of the worker based on one of the sets. That is, it is possible to cluster the feature values ​​Db of time-series motion extracted from the video 1Ba and arrange them in time series, and identify the periodic motion of the worker based on the frequent patterns identified from the series of clusters arranged in time series.

[0094] In addition, the periodic movements of a worker can be detected using either the first set or the second set, whichever has a higher degree of similarity between the elements that make up the set, thereby making it possible to improve the detection accuracy when detecting periodic movements.

[0095] Furthermore, since the start and end points of the periodic motion, the number of times the periodic motion occurs, and the evaluation content can be output, it is possible to easily recognize multiple motions of different lengths included in the worker's periodic motion and provide an evaluation of those motions.

[0096] Therefore, the periodic motion detection device 10 according to this embodiment can detect a periodic motion including a plurality of motions with different lengths.

[0097] § 4 Variations It should be noted that the present invention is not limited to the above-described embodiment, and can be embodied in various other forms without departing from the spirit and scope of the present invention. Therefore, the above-described embodiment is merely illustrative in all respects and should not be interpreted as being restrictive.

[0098] For example, in the above embodiment, the time-series information is a video, but the time-series information is not limited to a video. Specifically, the time-series information may be information on coordinate values ​​indicating the motion of the worker A measured by a motion capture device provided instead of the image sensor 20, or information indicating the motion of the worker A measured by having the worker A wear an acceleration sensor or a gyro sensor instead of the image sensor 20. The time-series information may be information indicating a change in pressure value measured by a pressure sensor arranged in the working area R instead of the image sensor 20, or information indicating a change in an event detected by a photoelectric sensor arranged in the working area R instead of the image sensor 20. Furthermore, the time-series information is not limited to using each of the above information individually, and may be a combination of two or more of the above information.

[0099] In addition, the embodiment of the present invention may be described as in the following appendices. However, the embodiment of the present invention is not limited to the form described in the following appendices. In addition, the embodiment of the present invention may be a form in which the descriptions between the appendices are replaced or combined.

[0100] [Appendix 1] a clustering unit (13) that classifies by clustering time-series movement feature quantities extracted from time-series information on the work of the worker, and outputs the classified clusters as one-dimensional time-series data; a labeling unit (14) that performs a labeling process of grouping consecutive identical clusters of the one-dimensional time series data into one cluster; a sequential pattern mining unit (16) for identifying a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time series data that has been subjected to the labeling process; a first frequent pattern extraction unit (17) that, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately after the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the end of the frequent pattern to be extracted; a second frequent pattern extraction unit (17) that, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the beginning of the frequent pattern to be extracted; a periodic behavior detection unit (18) that detects a periodic behavior based on either one of the set of frequent patterns extracted by the first frequent pattern extraction unit (17) or the set of frequent patterns extracted by the second frequent pattern extraction unit (17); A periodic motion detection device (10) comprising:

[0101] [Appendix 2] and an output unit (1A) that outputs the number of periodic operations, a start point of the periodic operation, and an end point of the periodic operation based on the periodic operation detected by the periodic operation detection unit (18). 2. The periodic motion detection device (10) of claim 1.

[0102] [Appendix 3] the periodic motion detection unit (18) compares an average value of the similarity between the frequent patterns included in the set of frequent patterns extracted by the first frequent pattern extraction unit (17) with an average value of the similarity between the frequent patterns included in the set of frequent patterns extracted by the second frequent pattern extraction unit (17), and detects a periodic motion based on the set with a higher average value of similarity. 3. The periodic motion detection device (10) of claim 1 or 2.

[0103] [Appendix 4] an evaluation unit (19) that evaluates a positional deviation of the detected periodic motion based on a degree of overlap between a detected periodic motion area in which the periodic motions detected by the periodic motion detection unit (18) are arranged in chronological order and an exemplary periodic motion area in which exemplary periodic motions are arranged in chronological order, The output unit (1A) further outputs the content evaluated by the evaluation unit (19). A periodic motion detection device (10) according to any one of appendix 1 to 3.

[0104] [Appendix 5] an exclusion unit (15) that excludes a first cluster and a last cluster as excluded clusters from the one-dimensional time series data that has been subjected to the labeling process; the sequential pattern mining unit (16) identifies, as a frequent pattern, a sequence of frequently occurring clusters from the one-dimensional time series data from which the excluded cluster has been excluded. A periodic motion detection device (10) according to any one of appendix 1 to 4.

[0105] [Appendix 6] A method executed by a processor (10a), comprising: a clustering step of classifying by clustering time-series movement feature quantities extracted from the time-series information on the work of the worker, and outputting the classified clusters as one-dimensional time-series data; a labeling step of performing a labeling process of grouping multiple consecutive identical clusters into one cluster in the one-dimensional time series data; a sequential pattern mining step of identifying a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time series data that has been subjected to the labeling process; a first frequent pattern extraction step of extracting the identified frequent pattern from the one-dimensional time series data by sequentially extracting a series of clusters constituting a series of infrequent patterns located immediately after the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the end of the frequent pattern to be extracted; a second frequent pattern extraction step of extracting the identified frequent pattern from the one-dimensional time series data by linking a sequence of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted to the beginning of the frequent pattern to be extracted, and sequentially extracting the frequent pattern as one of the frequent patterns; a periodic action detection step of detecting a periodic action based on either one of the set of the frequent patterns extracted in the first frequent pattern extraction step or the set of the frequent patterns extracted in the second frequent pattern extraction step; A periodic motion detection method comprising:

[0106] [Appendix 7] Computer, a clustering unit (13) that classifies by clustering time-series movement feature quantities extracted from time-series information on the work of the worker, and outputs the classified clusters as one-dimensional time-series data; a labeling unit (14) that executes a labeling process for grouping consecutive identical clusters into one cluster in the one-dimensional time series data; a sequence pattern mining unit (16) for identifying a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time series data that has been subjected to the labeling process; a first frequent pattern extraction unit (17) which, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately after the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the end of the frequent pattern to be extracted; a second frequent pattern extraction unit (17) which, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a sequence of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted as one frequent pattern by linking the sequence of clusters to the beginning of the frequent pattern to be extracted; a periodic behavior detection unit (18) that detects a periodic behavior based on either the set of frequent patterns extracted by the first frequent pattern extraction unit (17) or the set of frequent patterns extracted by the second frequent pattern extraction unit (17); A periodic motion detection program that acts as a [Explanation of symbols]

[0107] 10...periodic motion detection device, 10a...CPU, 10b...RAM, 10c...ROM, 10d...communication device, 10e...input device, 10f...display device, 20...image sensor, 100...system, A...worker, R...work area, 11...acquisition unit, 12...extraction unit, 13...clustering unit, 14...labeling unit, 15...exclusion unit, 16...sequential pattern mining unit, 17...frequent pattern extraction unit, 18...periodic motion detection unit, 19...evaluation unit, 1A...output unit, 1B...memory unit, 1Ba...video, 1Bb...trained model

Claims

1. a clustering unit that classifies by clustering time-series movement feature quantities extracted from time-series information on the work of the worker, and outputs the classified clusters as one-dimensional time-series data; a labeling unit that performs a labeling process of grouping a plurality of consecutive identical clusters into one cluster in the one-dimensional time series data; a sequential pattern mining unit that identifies a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time series data that has been subjected to the labeling process; a first frequent pattern extraction unit that, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately after the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the end of the frequent pattern to be extracted; a second frequent pattern extraction unit that, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the beginning of the frequent pattern to be extracted; a periodic action detection unit that detects a periodic action based on either one of the set of the frequent patterns extracted by the first frequent pattern extraction unit or the set of the frequent patterns extracted by the second frequent pattern extraction unit; A periodic motion detection device comprising:

2. and an output unit that outputs the number of occurrences of the periodic operation, a start point of the periodic operation, and an end point of the periodic operation based on the periodic operation detected by the periodic operation detection unit.

2. The periodic motion detector of claim 1.

3. the periodic action detection unit compares an average value of similarity between the frequent patterns included in the set of the frequent patterns extracted by the first frequent pattern extraction unit with an average value of similarity between the frequent patterns included in the set of the frequent patterns extracted by the second frequent pattern extraction unit, and detects a periodic action based on the set having a higher average value of similarity.

3. The periodic motion detection device according to claim 1 or 2.

4. an evaluation unit that evaluates a position shift of the detected periodic motion based on a degree of overlap between a detected periodic motion area in which the periodic motions detected by the periodic motion detection unit are arranged in chronological order and an exemplary periodic motion area in which exemplary periodic motions are arranged in chronological order, The output unit further outputs the content evaluated by the evaluation unit.

3. The periodic motion detector of claim 2.

5. An exclusion unit that excludes a first cluster and a last cluster as excluded clusters from the one-dimensional time series data that has been subjected to the labeling process, the sequential pattern mining unit identifies, as a frequent pattern, a sequence of frequently occurring clusters from the one-dimensional time series data from which the excluded cluster has been excluded. The periodic motion detection device according to claim 1 .

6. 1. A processor-implemented method, comprising: a clustering step of classifying by clustering time-series movement feature quantities extracted from time-series information on the work of the worker, and outputting the classified clusters as one-dimensional time-series data; a labeling step of performing a labeling process of grouping a plurality of consecutive identical clusters into one cluster in the one-dimensional time series data; a sequential pattern mining step of identifying a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time series data that has been subjected to the labeling process; a first frequent pattern extraction step of extracting the identified frequent pattern from the one-dimensional time series data by sequentially extracting a series of clusters constituting a series of infrequent patterns located immediately after the frequent pattern to be extracted, as one frequent pattern by linking the series of clusters to the end of the frequent pattern to be extracted; a second frequent pattern extraction step of extracting the identified frequent pattern from the one-dimensional time series data by sequentially extracting a series of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the beginning of the frequent pattern to be extracted; a periodic motion detection step of detecting a periodic motion based on either one of the set of the frequent patterns extracted in the first frequent pattern extraction step or the set of the frequent patterns extracted in the second frequent pattern extraction step; A periodic motion detection method comprising:

7. Computer, a clustering unit that classifies by clustering time-series movement feature quantities extracted from time-series information on the work of the worker, and outputs the classified clusters as one-dimensional time-series data; a labeling unit that performs a labeling process of grouping a plurality of consecutive identical clusters into one cluster in the one-dimensional time series data; a sequence pattern mining unit that identifies a sequence of frequently occurring clusters as a frequent pattern from the one-dimensional time series data that has been subjected to the labeling process; a first frequent pattern extraction unit which, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately after the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the end of the frequent pattern to be extracted; a second frequent pattern extraction unit which, when extracting the identified frequent pattern from the one-dimensional time series data, sequentially extracts a series of clusters constituting a series of infrequent patterns located immediately before the frequent pattern to be extracted as one frequent pattern by linking the series of clusters to the beginning of the frequent pattern to be extracted; a periodic action detection unit that detects a periodic action based on either one of the set of the frequent patterns extracted by the first frequent pattern extraction unit or the set of the frequent patterns extracted by the second frequent pattern extraction unit; A periodic motion detection program that acts as a

Images