Work cycle estimation device, work cycle estimation method, and work cycle estimation program

Abstract

A work cycle estimation device is provided with: an activity information acquisition unit that acquires activity information relating to activities when a worker performs a series of predetermined works; an estimation unit that estimates a start point and an end point of a work cycle of the worker on the basis of the activity information; and an output unit that outputs the estimated start point and end point of the work cycle.

Description

Technical Field

[0001] The disclosed technology relates to an operation cycle estimation device, an operation cycle estimation method and an operation cycle estimation program. Background Art

[0002] In Japanese Patent Gazette No. 2022-3491, the following analysis device is disclosed: extracting a first theme and a second theme from time series data showing the actions of an operator repeating a first process, respectively tracking the first theme and the second theme that appear repeatedly in the time series data, and using the respective intervals between the tracked multiple first themes and multiple second themes to analyze the time outside the first process in the time series data. Summary of the invention

[0003] Technical problem to be solved by the invention

[0004] However, the technology described in Japanese Patent Application Laid-Open No. 2022-3491 has a problem in that since a subject is tracked using a particle filter based on a standard working time input in advance, the standard working time needs to be measured in advance.

[0005] The disclosed technology is completed in view of the above points, and its purpose is to provide a work cycle estimation device, work cycle estimation method and work cycle estimation program that can estimate the starting point and end point of the work cycle without measuring the standard work time in advance.

[0006] Solutions for solving technical problems

[0007] The first disclosed method is a work cycle estimation device, comprising: an activity information acquisition unit, which acquires activity information related to the activities of an operator when performing a predetermined series of operations; an estimation unit, which estimates the starting point and end point of the operator's work cycle based on the activity information; and an output unit, which outputs the estimated starting point and end point of the work cycle.

[0008] The second disclosed method is that in the above-mentioned first method, the estimation unit includes: a posture sequence generating unit, which generates a posture sequence related to the change of the posture based on the skeleton of the operator based on the activity information, and the posture sequence removes the inherent information of the operator; a segment sequence generating unit, which generates a segment sequence according to each determined posture, and the segment sequence divides the posture sequence into multiple segments in a manner of including a posture determined once; and a setting unit, which sets the beginning and end of each segment contained in the segment sequence with the highest similarity between the multiple segment sequences generated according to each of the determined postures as the starting point and end point of each work cycle.

[0009] The third method disclosed is that in the above-mentioned second method, it can also be constructed as follows: the posture sequence generation unit includes: a motion vector sequence calculation unit, which calculates the motion vector sequence of the operator based on the activity information; and a clustering unit, which clusters the motion vector sequence and generates a cluster sequence, and the segment sequence generation unit includes: a first segment sequence generation unit, which generates a first segment sequence for each of a plurality of clusters, the first segment sequence starts with a cluster selected from the plurality of clusters and divides the cluster sequence into a plurality of first segments; a frequently occurring pattern determination unit, which determines, for each of the plurality of first segment sequences generated, A frequently occurring pattern of clusters common among a plurality of said first segments is determined; and a second segment sequence generating unit is configured to generate a second segment sequence, wherein said second segment sequence divides said cluster sequence into a plurality of second segments including said frequently occurring pattern, said setting unit comprising: an evaluation value calculating unit, which calculates, for each of said plurality of second segment sequences generated, an evaluation value related to the similarity between said plurality of second segments; and a determining unit, which determines, as the starting point and the end point of each operation cycle, the respective beginning and the end of the second segment included in the second segment sequence having the highest evaluation value among the evaluation values ​​calculated for each of said plurality of second segment sequences.

[0010] A fourth disclosed aspect is that, in the third aspect, the clustering unit removes from the cluster sequence clusters that exist only near the beginning or the end of the cluster sequence, that is, clusters whose ratio to the length of the cluster sequence is less than a predetermined first threshold.

[0011] The fifth method disclosed is that in the above-mentioned third method or fourth method, the second segment sequence generation unit uses 1 / 2 of the cycle length of multiple second segments contained in the second segment sequence as the second threshold, and combines the second segment with a cycle length less than the second threshold with the previous second segment.

[0012] A sixth disclosed aspect is, in any one of the third to fifth aspects, wherein the evaluation value calculation unit excludes second segments, for which all similarities between a plurality of second segments are equal to or less than a third threshold value determined in advance, from calculation targets of the evaluation value.

[0013] The seventh method disclosed, in any one of the third to sixth methods mentioned above, may also be: the first segment sequence generating unit generates an end-exclusion segment sequence and a beginning-exclusion segment sequence, the end-exclusion segment sequence excludes the first segment at the end, and the beginning-exclusion segment sequence excludes the first segment at the beginning, and the frequent occurrence pattern determining unit determines the frequent occurrence pattern obtained by combining the frequent occurrence pattern of the end-exclusion segment sequence and the frequent occurrence pattern of the beginning-exclusion segment sequence as the frequent occurrence pattern of the cluster common to multiple first segments.

[0014] The eighth disclosed method is that in any one of the third to seventh methods mentioned above, the second segment sequence generating unit may generate: a second segment sequence obtained by dividing the clustering sequence into a second segment in which a pattern other than the second segment candidate including the frequently occurring pattern is appended to the beginning of the second segment candidate adjacent to the pattern, and a second segment sequence obtained by dividing the clustering sequence into a second segment in which a pattern other than the second segment candidate is appended to the end of the second segment candidate adjacent to the pattern.

[0015] A ninth disclosed aspect is, in any one of the second to eighth aspects, wherein the posture sequence is a sequence based on the postures of the skeletons of the upper body and lower body of the worker.

[0016] A tenth disclosed aspect is, in any one of the second to ninth aspects, wherein the posture sequence generation unit generates a posture sequence from which the influence of the camera viewpoint is eliminated.

[0017] The eleventh disclosed method is that in any one of the above-mentioned second to tenth methods, the posture sequence generation unit generates a skeleton sequence of the operator based on the activity information, and performs at least one of interpolation processing and smoothing processing on the generated skeleton sequence, and then generates the posture sequence.

[0018] The twelfth disclosed method is a work cycle estimation method, in which a computer executes the following processing: obtaining activity information related to the activities of an operator when performing a predetermined series of operations; estimating the starting point and end point of the operator's work cycle based on the activity information; and outputting the estimated starting point and end point of the work cycle.

[0019] The thirteenth method disclosed is a work cycle estimation program, which enables a computer to execute the following processing: obtaining activity information related to the activities of an operator when performing a predetermined series of operations; estimating the starting point and end point of the operator's work cycle based on the activity information; and outputting the estimated starting point and end point of the work cycle.

[0020] Effects of the Invention

[0021] According to the disclosed technology, the start point and the end point of a work cycle can be estimated without measuring the standard work time in advance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the structural diagram of the operation cycle estimation system.

[0023] Figure 2 : is a block diagram showing the hardware structure of the working period estimation device.

[0024] Figure 3 This is a functional block diagram of a work cycle estimation device.

[0025] Figure 4 This is a diagram showing an example of a clustering sequence.

[0026] Figure 5 It is a diagram for explaining the first sequence.

[0027] Figure 6 It is a diagram for explaining the second sequence.

[0028] Figure 7 It is a diagram for explaining the second sequence.

[0029] Figure 8 This is a diagram for explaining the start and end points of a work cycle.

[0030] Fig. 9 This is a flowchart of the work cycle estimation process. DETAILED DESCRIPTION

[0031] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the accompanying drawings. In addition, in each of the drawings, the same reference numerals are given to the same or equivalent components and parts. In addition, the dimensional ratios of the drawings are sometimes exaggerated for the sake of convenience of description and sometimes differ from the actual ratios.

[0032] Figure 1 The following is a diagram showing a configuration of a work cycle estimation system 10 . The work cycle estimation system 10 includes a work cycle estimation device 20 and a camera 30 .

[0033] The work cycle estimation device 20 estimates the start and end points of the work cycle of the work performed by the worker W based on the moving image captured by the camera 30 .

[0034] For example, the worker W takes out the work object M placed on the work table TB and performs a series of predetermined operations in the work space S. The series of operations performed by the worker W are operations including a plurality of actions in one operation cycle, such as holding, transporting, assembling, inspecting, screwing with a driver, and labeling of parts.

[0035] The camera 30 is, for example, an imaging device capable of capturing RGB color moving images. The camera 30 is installed at a position where the movement of the worker W and the entire workbench TB can be easily recognized.

[0036] In addition, in the present embodiment, a case where one camera 30 is provided has been described, but a configuration in which a plurality of cameras 30 are provided may also be adopted.

[0037] Figure 22 is a block diagram showing the hardware structure of the operation cycle estimation device 20 according to the present embodiment. Figure 2 As shown, the operation cycle estimation device 20 includes a controller 21. The controller 21 is composed of a device including a general computer.

[0038] like Figure 2 As shown, the controller 21 includes a CPU (Central Processing Unit) 21A, a ROM (Read Only Memory) 21B, a RAM (Random Access Memory) 21C, and an input / output interface (I / O) 21D. The CPU 21A, the ROM 21B, the RAM 21C, and the I / O 21D are connected via a bus 21E. The bus 21E includes a control bus, an address bus, and a data bus.

[0039] Furthermore, an operation section 22 , a display section 23 , a communication section 24 , and a storage section 25 are connected to the I / O 21D.

[0040] The operation unit 22 is configured to include, for example, a mouse and a keyboard.

[0041] The display unit 23 is composed of, for example, a liquid crystal display or the like.

[0042] The communication unit 24 is an interface for performing data communication with an external device such as a camera 30 .

[0043] The storage unit 25 is composed of a non-volatile external storage device such as a hard disk. Figure 2 As shown, the storage unit 25 stores a work cycle estimation program 25A and the like.

[0044] CPU21A is an example of a computer. The computer mentioned here refers to a processor in a broad sense, including a general-purpose processor (e.g., CPU) or a dedicated processor (e.g., GPU: Graphics Processing Unit, graphics processor, ASIC: Application Specific Integrated Circuit, dedicated integrated circuit, FPGA: Field Programmable Gate Array, field programmable gate array, programmable logic device, etc.).

[0045] In addition, the operation period estimation program 25A may be realized by being stored in a non-volatile non-transitory recording medium or distributed via a network and then appropriately installed in the operation period estimation device 20 .

[0046] Examples of nonvolatile non-migratable recording media include CD-ROM (Compact Disc Read Only Memory), magneto-optical disk, HDD (Hard Disk Drive), DVD-ROM (Digital Versatile Disc Read Only Memory), flash memory, memory card, etc.

[0047] Figure 3 2 is a block diagram showing the functional structure of the CPU 21A of the operation cycle estimation device 20. Figure 3 As shown, the CPU 21A functionally includes a motion information acquisition unit 40 , an estimation unit 41 , and an output unit 42 .

[0048] The CPU 21A reads and executes the operation period estimation program 25A stored in the storage unit 25 as a Figure 3 The various functional parts shown in the figure perform their functions.

[0049] The activity information acquisition unit 40 acquires activity information related to the activity of the worker W when performing a series of predetermined operations. Specifically, a dynamic image obtained by photographing the worker W performing a series of predetermined operations by the camera 30 is stored in advance in the storage unit 25 as activity information. Then, the activity information acquisition unit 40 acquires the activity information by reading the dynamic image stored in the storage unit 25. In addition, the activity information of the worker W may be acquired by using a motion sensor or the like instead of the camera 30.

[0050] The estimation unit 41 estimates the start and end points of the operation cycle of the operator W based on the dynamic image acquired by the activity information acquisition unit 40. Here, the operation cycle refers to the cycle of a series of operations repeatedly performed by the operator W. Moreover, the start point of the operation cycle refers to the time when the series of operations starts, and the end point of the operation cycle refers to the time when the series of operations ends.

[0051] The estimation unit 41 includes a posture sequence generation unit 41A, a segment sequence generation unit 41B, and a setting unit 41C.

[0052] The posture sequence generation unit 41A generates a posture sequence related to changes in posture of the skeleton of the worker W, that is, a posture sequence from which unique information of the worker W is removed, based on the dynamic image acquired by 40 . The posture sequence generation unit 41A includes a motion vector sequence calculation unit 43 and a clustering unit 44 .

[0053] The segment sequence generation unit 41B generates a segment sequence for each determined posture (key action) by dividing the posture sequence generated by the posture sequence generation unit 41A into a plurality of segments so as to include the posture determined once. The segment sequence generation unit 41B includes a first segment sequence generation unit 45, a frequently appearing pattern determination unit 46, and a second segment sequence generation unit 47.

[0054] The setting unit 41C sets the beginning and the end of each segment included in the segment sequence with the highest similarity among the plurality of segment sequences generated by the segment sequence generation unit 41B for each determined posture as the start point and the end point of each work cycle. The setting unit 41C includes an evaluation value calculation unit 48 and a determination unit 49.

[0055] The motion vector sequence calculation unit 43 calculates the motion vector sequence of the worker W based on the dynamic image acquired by the motion information acquisition unit 40. The motion vector sequence is an example of a posture sequence, and is sequence data of motion feature quantities obtained by applying motion feature quantity extraction processing to a skeleton sequence.

[0056] Specifically, the motion vector sequence calculation unit 43 estimates the posture of the worker W based on the moving image so as not to be affected by the background and inherent information such as clothing of the worker W, and converts the estimated posture into a skeleton sequence.

[0057] As a method for estimating the posture of the operator W and converting the estimated posture into a skeleton sequence, a well-known method called OpenPose (Real-time Multi-Person Posture Estimation Library) described in the following reference 1 can be used. The skeleton sequence is a time series of data including the coordinates of feature points such as body parts and joints of the operator W, and labels of body parts representing the feature points. For example, the feature points include the parts of the face such as the eyes and nose of the operator W, and the joints such as the neck, shoulders, elbows, wrists, waist, knees, and ankles.

[0058] In OpenPose, a learned model is used that uses multiple dynamic images as teacher data to learn a learning model that uses dynamic images as input and a skeleton sequence as output. As a learning method for obtaining such a learned model, a well-known method such as CNN (Convolutional Neural Networks) is used.

[0059] (Reference 1) "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", Zhe Cao, Student Member, IEEE, Gines Hidalgo, Student Member, IEEE, Tomas Simon, Shih-En Wei, and Yaser Sheikh, IEEE TRANSACTIONS ONPATTERN ANALYSIS AND MACHINE INTELLIGENCE.

[0060] Here, in the factory operation, since various operators with different body shapes perform the operation, the influence caused by different body shapes is relatively large. In order to avoid the influence of different body shapes, in this embodiment, a technique called Motion Retargeting described in the following reference 2 is used to convert the skeleton sequence obtained in the dynamic image into a motion vector sequence representing the motion feature quantity.

[0061] (Reference 2) K. Aberman, R. Wu, D. Lischinski, B. Chen, and D. Cohen-Or, "Learning character-agnostic motion for motion retargeting in 2d," TOG, vol.38, no.4, p.75, 2019.

[0062] In Motion Retargeting, a skeleton sequence is taken as input and an encoder is used to output feature vectors of three components, namely, motion, body shape, and camera viewpoint, i.e., motion feature quantities. However, in this embodiment, as an example, in order to eliminate the influence of body shape and camera viewpoint, only the feature vector of the motion component is used.

[0063] Furthermore, in order to eliminate the influence of noise on posture estimation, at least one of the three pre-processing processes of time series interpolation, time series smoothing, and lower body interpolation may be performed on the skeleton sequence before conversion to a motion vector sequence.

[0064] In the interpolation process of the time series, if there are joints where the posture estimation fails, the joints of the previous frame are copied. In the time series smoothing process, in order to remove the noise of the posture estimation, the sequence data is smoothed using a Gaussian filter. In the OpenPose used in this embodiment, not only the upper body of the character is estimated, but also the posture of the lower body is estimated. That is, a sequence of postures based on the bones of the upper body and lower body of the operator W is estimated. In the case of working in a factory, since most of the work is done using a workbench on a table, the lower body is blocked by the table and most of the joints of the lower body are missing. Since the encoder that extracts the motion feature value such as Motion Retargeting takes the whole body skeleton sequence of the character as input, when the joints of the lower body are missing, the feature vector of the motion component may not be output properly. Therefore, the interpolation process of the lower body can also be performed. Specifically, as the interpolation process of the lower body, the joints of at least one of the two knees and the two feet can also be interpolated with a length proportional to the length of the body of the character.

[0065] The clustering unit 44 generates a cluster sequence by clustering the motion vector sequence calculated by the motion vector sequence calculation unit 43. Here, the cluster sequence refers to one-dimensional (cluster number) sequence data obtained as a result of applying clustering to the motion vector sequence.

[0066] Specifically, the clustering unit 44 converts the high-dimensional motion vector sequence into a one-dimensional cluster sequence. In this embodiment, as an example, spectral clustering is applied to convert the high-dimensional motion vector sequence calculated by the motion vector sequence calculation unit 43 into a one-dimensional cluster sequence. Spectral clustering is an unsupervised clustering technique. By clustering the motion vector sequence, similar motions are classified into the same cluster, so it is easy to capture periodicity.

[0067] In factory operations, at the beginning and end of the operations, scenes other than the assembly of the product to be analyzed may sometimes occur. Specifically, these are the waiting time before assembly and the waiting time at the end of the operation. Since these scenes become noise when estimating the starting point and end point of the operation cycle, they can also be removed in pre-processing. For example, as a pre-processing, a process is performed to remove clusters that only exist near the beginning or end of the cluster sequence, that is, clusters whose ratio to the length of the cluster sequence is less than a predetermined first threshold. The first threshold can be, for example, a length of 0.05 times the data length of the cluster sequence. In addition, the data length of the cluster sequence refers to the number of clusters contained in the cluster sequence. For example, when the data length of the cluster sequence is 100, the range of five quantities from the beginning is near the beginning, and the range of five quantities from the end is near the end.

[0068] Figure 4 An example of a clustering sequence is shown. Figure 4 The cluster sequence C1 shown is a cluster sequence in which motion vector sequences are clustered when the worker W performs a series of operations a plurality of times. The motions of the worker W are clustered into six clusters 0 to 5.

[0069] Here, cluster 5 is a cluster that exists only at the beginning. Since the ratio to the length of the cluster sequence is less than 0.05, the movement of cluster 5 is noise, and the data of cluster 5 is removed from cluster sequence C1 to obtain cluster sequence C2.

[0070] In addition, when a beginner performs an operation, the operation time of each operation cycle may be different. However, in order to determine the operation cycle, it is important to focus only on the order of the operations that appear. Therefore, processing is required to avoid being affected by the operation time of each operation cycle. Therefore, the one-dimensional cluster sequence C2 that has been noise-removed in the pre-processing is subjected to labeling processing to simplify the continuous data of the same cluster.

[0071] exist Figure 4 In the example shown, the cluster sequence C2 is 0, 1, 0, 2, 2, 3, 3, 4, ..., and the values ​​of cluster 2 and cluster 3 are continuous. Such continuous values ​​are aggregated into one, and a simplified cluster sequence C3 is generated such as 0, 1, 0, 2, 3, 4, ... However, in the case of estimating the start point and the end point of the operation cycle, since it is necessary to estimate the start point and the end point of each operation cycle in the dynamic image, the cluster sequence C2 before simplification is stored in the storage unit 25 in advance.

[0072] In factory operations, there are often inherent actions that occur only once in a cycle, such as product assembly. For example, the action of taking out a special component and returning the finished product to a storage box only occur once in a cycle. In this embodiment, the action that occurs only once in a cycle is defined as a key activity, and the first sequence is generated based on the cluster sequence C3 with an eye on the key activity.

[0073] The first segment sequence generating unit 45 generates a first segment sequence for each of the plurality of clusters, the first segment sequence starting with the cluster selected as the key action from the plurality of clusters clustered by the clustering unit 44, and dividing the cluster sequence C2 into a plurality of first segments. That is, each of the plurality of clusters is sequentially selected as the key action, and a first segment sequence is generated for each. Here, the first segment sequence refers to cluster sequence data obtained by dividing the cluster sequence simplified in the time series direction in the labeling process by the key action.

[0074] exist Figure 4In the example, since cluster sequence C3 is five clusters from cluster 0 to cluster 4, each of the five clusters is selected as a key action in turn to generate the first sequence.

[0075] Specifically, cluster 0 is first selected as the key action, and cluster sequence C3 is segmented starting with cluster 0. Figure 5 As shown, the cluster sequence C3 is divided into six first segments A01 to A06, namely "0, 1", "0, 2, 3, 4", "0, 1, 2, 3, 4", "0, 1, 3, 4", "0, 1", and "0, 3, 4", generating a first segment sequence A0 consisting of these six first segments A01 to A06.

[0076] Next, cluster 1 is selected as the key action, and cluster sequence C3 is segmented starting with cluster 1. Thus, cluster sequence C3 is segmented into five first segments A11 to A15, namely "0", "1, 0, 2, 3, 4, 0", "1, 2, 3, 4, 0", "1, 3, 4, 0", and "1, 0, 3, 4", and a first segment sequence A1 consisting of these five first segments A11 to A15 is generated. Similarly, first segment sequences A2 to A4 are generated for clusters 2 to 4, respectively.

[0077] The frequent appearance pattern identification unit 46 identifies a frequent appearance pattern of clusters common among the plurality of first segments for each of the plurality of first segment sequences A0 to A4 generated by the first segment sequence generation unit 45 .

[0078] Frequently occurring patterns can be identified using, for example, a technique described in Reference 3 below called PrefixSpan (prefix projection pattern mining), which is a sequential pattern mining technique.

[0079] (Reference 3) J.Han, J.Pei, B.Mortazavi-Asl, H.Pinto, Q.Chen, U.Dayal, andM.Hsu, "Prefixspan: Mining sequential patterns efficiently by prefix-projected pattern growth," ICDEIEEE, pp.215-224 2001.

[0080] The frequent occurrence pattern determination unit 46 estimates a frequent occurrence pattern common to a plurality of first segments using PrefixSpan, and specifically, uses the pattern with the highest number of appearances among the frequent occurrence pattern candidates as the final frequent occurrence pattern.

[0081] Here, the frequently occurring pattern identification unit 46 may not apply PrefixSpan to all of the first segments constituting the first segment sequence, but may apply sequence pattern mining based on PrefixSpan to the first segment sequence excluding the last first segment and the first segment sequence excluding the first first segment.

[0082] This is to prevent, when the key action appears near the center of each work cycle, any one of the pattern appearing before the key action and the pattern appearing after the key action from being estimated as a frequently appearing pattern.

[0083] There are three locations where key actions appear: the beginning, the end, and other than the beginning and the end of the work cycle. When a key action appears at the beginning or the end, by applying sequence pattern mining to the first segment obtained by segmenting the cluster sequence at the position of the key action, it is easy to extract the frequently occurring patterns covering each work cycle.

[0084] However, if sequential pattern mining is applied to the first sequence obtained by segmenting the clustered sequence at the position of the key action that appears except at the beginning and the end, only the frequent occurrence pattern before the key action or the frequent occurrence pattern after the key action is extracted. Therefore, the frequent occurrence pattern before the key action can be extracted from the first sequence obtained by removing the end of the first sequence, and the frequent occurrence pattern after the key action can be extracted from the segment removing the beginning, and these frequent occurrence patterns can be combined.

[0085] That is, it may also be that the first segment sequence generating unit 45 generates an end exclusion segment sequence excluding the end of the first segment, and a beginning exclusion segment sequence excluding the beginning of the first segment, and the frequent occurrence pattern determining unit 46 determines the frequent occurrence pattern obtained by combining the frequent occurrence pattern of the end exclusion segment sequence and the frequent occurrence pattern of the beginning exclusion segment sequence as the frequent occurrence pattern of the cluster common among multiple first segments.

[0086] Thus, even when the key action appears at other than the beginning and the end of the work cycle, it is possible to extract the frequently appearing pattern with high accuracy.

[0087] exist Figure 5In the example, since the first segment sequence A0 in cluster 0 is composed of six first segments A01 to A06, "0" is determined as the frequent occurrence pattern of the end exclusion segment sequence A0A composed of the first segments A01 to A05 excluding the end first segment A06. In addition, "0" is determined as the frequent occurrence pattern of the head exclusion segment sequence A0B composed of the first segments A02 to A06 excluding the head first segment A01. In this case, since the frequent occurrence pattern of the end exclusion segment sequence A0A is the same as the frequent occurrence pattern of the head exclusion segment sequence A0B, "0" is determined as the frequent occurrence pattern of the first segment sequence A0.

[0088] Similarly, "0" is determined as the frequent occurrence pattern of the end exclusion segment sequence A1A consisting of the first segments A11 to A14 excluding the end first segment A15 from the first segment sequence A1 in cluster 1. In addition, "3, 4" is determined as the frequent occurrence pattern of the head exclusion segment sequence A1B consisting of the first segments A12 to A15 excluding the head first segment A11. In this case, since the frequent occurrence pattern of the end exclusion segment sequence A1A is different from the frequent occurrence pattern of the head exclusion segment sequence A1B, "0, 3, 4" which is a combination of "0" and "3, 4" is determined as the frequent occurrence pattern of the first segment sequence A1.

[0089] Next, the frequently occurring patterns are determined for the first sequences in clusters 2 to 4 respectively.

[0090] The second segment sequence generation unit 47 generates a second segment sequence that divides the cluster sequence into a plurality of second segments including the frequently occurring pattern. Specifically, the second segment sequence generation unit 47 generates: a second segment sequence that divides the cluster sequence into a second segment in which a pattern other than the second segment candidate including the frequently occurring pattern is added to the beginning of the second segment candidate adjacent to the pattern, and a second segment sequence that divides the cluster sequence into a second segment in which a pattern other than the second segment candidate is added to the end of the second segment candidate adjacent to the pattern.

[0091] More specifically, first, the set of clusters including the frequently occurring pattern is searched sequentially from the beginning of the cluster sequence C3. Then, the set of clusters whose ratio of the number of clusters constituting the frequently occurring pattern to the number of clusters constituting the set of clusters including the frequently occurring pattern is greater than a predetermined threshold is used as the second segment candidate. In addition, as an example, the threshold can be set to 0.2, but is not limited thereto.

[0092] As mentioned above, the frequently occurring pattern in the first sequence A0 is "0". Figure 6 As shown, five frequently occurring patterns "0" indicated by bold frames are retrieved from the beginning of the cluster sequence C3. The frequently occurring pattern "0" at the beginning becomes the second segment B01.

[0093] Next, a pattern obtained by adding a pattern other than the frequently occurring pattern "0" to the adjacent frequently occurring pattern "0" is used as the second segment. Figure 6 , the pattern between the first frequently occurring pattern "0" and the second frequently occurring pattern "0" is "1". Therefore, the pattern "1, 0" obtained by appending the pattern "1" to the second frequently occurring pattern "0" adjacent thereto becomes the second segment B02. In addition, the pattern between the second frequently occurring pattern "0" and the third frequently occurring pattern "0" is "2, 3, 4". Therefore, the pattern "2, 3, 4, 0" obtained by appending the pattern "2, 3, 4" to the third frequently occurring pattern "0" adjacent thereto becomes the second segment B03. Similarly, the second segments B04 to B06 are generated below, and the second segment sequence B0 including the second segments B01 to B06 is generated. In addition, the "3, 4" at the end is the remaining pattern that does not become the second segment.

[0094] Next, generate a second sequence of other patterns. Specifically, the pattern "0, 1" obtained by appending the pattern "1" between the first frequent occurrence pattern "0" and the second frequent occurrence pattern "0" to the first frequently occurring pattern "0" adjacent to the previous one is used as the second segment b01. In addition, the pattern "0, 2, 3, 4" obtained by appending the pattern "2, 3, 4" between the second frequent occurrence pattern "0" and the third frequent occurrence pattern "0" to the second frequently occurring pattern "0" adjacent to the previous one is used as the second segment b02. Similarly, generate the second segments b03 to b06 and generate the second sequence b0.

[0095] In this way, for the first sequence A0 in cluster 0, two patterns of second sequences B0 and b0 are generated.

[0096] In addition, as mentioned above, the frequently occurring pattern in the first sequence A1 is "0, 3, 4". Figure 7 As shown, four patterns including the frequently occurring patterns "0, 3, 4" shown in bold frames are retrieved from the beginning of the cluster sequence C3.

[0097] Then, there is no pattern between the first "0, 2, 3, 4" and the second pattern "0, 1, 2, 3, 4". Figure 7 In the example, since the leading pattern "0, 1" does not include a frequently occurring pattern, "0, 1, 0, 2, 3, 4" is added to the subsequent pattern "0, 2, 3, 4" as the second segment B11. Similarly, the second segments B12 to B14 are generated, and the second segment sequence B1 is generated.

[0098] Next, the second segment sequence of other patterns is generated. Specifically, since the pattern including the frequently occurring pattern does not exist before the first pattern "0, 1", it cannot be attached to the previous pattern, so the pattern "0, 2, 3, 4" becomes the first second segment b11. Similarly, the second segments b12 to b14 are generated, and the second segment sequence b1 is generated. In addition, the first "0, 1" is the remaining pattern that does not become the second segment.

[0099] In this way, for the first sequence A1 in cluster 1, two patterns of second sequences B1 and b1 are generated.

[0100] Similarly, for the first segment sequence A2 in cluster 2, two patterns of second segment sequences B2 and b2 are generated, for the first segment sequence A3 in cluster 3, two patterns of second segment sequences B3 and b3 are generated, and for the first segment sequence A4 in cluster 4, two patterns of second segment sequences B4 and b4 are generated. Thus, ten second segment sequences are generated.

[0101] In addition, in order to remove noise, two post-processings may be performed on the generated second segment sequence. The first post-processing is to remove noise focusing on the lengths of the multiple second segments included in the second segment sequence. Here, the length of the second segment refers to the number of clusters constituting the second segment, hereinafter referred to as the cycle length. The first post-processing is to use 1 / 2 of the cycle length of the multiple second segments included in the second segment sequence as the second threshold, and to combine the second segment whose cycle length is less than the second threshold with the previous second segment.

[0102] The second post-processing is to remove noise by focusing on the similarity between the multiple second segments contained in the second segment sequence. For all combinations of multiple second segments, DTW (Dynamic Time Warping) with Hamming distance as the distance function is calculated as the similarity, and the second segments whose calculated similarity is all below a predetermined third threshold are deleted. In addition, the threshold is a value of 1 / 2 of the median (central value) of the above similarity.

[0103] The evaluation value calculation unit 48 calculates an evaluation value related to the similarity between the plurality of second segments for each of the plurality of second segment sequences generated by the second segment sequence generation unit 47 .

[0104] The evaluation value calculation unit 48 calculates the evaluation value S using the following formula (1) as an example.

[0105] S=S s w s +S c w c +S l w l

[0106] ···(1)

[0107] Here, S s The calculation is performed using the following formula (2).

[0108]

[0109] Here, C represents a combination. In addition, N is the number of second segments included in the second segment sequence. In addition, n1 and n2 are indices representing the second segments. n1n2 is the Kronecker function for n1 and n2. In addition, R n1 , R n2 Indicates the two second segments for calculating similarity. dtw(R n1 , R n2 ) is the DTW with the aforementioned Hamming distance as the distance function, which is used to calculate R n1 , R n2 The similarity function.

[0110] For example, when the second segment sequence includes three second segments, the similarity between the first second segment and the second second segment, the similarity between the second second segment and the third second segment, and the similarity between the first second segment and the third second segment are calculated respectively, and the average value of the calculated similarities becomes the evaluation value S. s .

[0111] In addition, S c The following formula (3) is used for calculation. In addition, as mentioned above, the beginning or end of the second sequence may contain a pattern that does not become the second segment. Therefore, the following formula (3) is used to calculate the ratio of the clusters that constitute the remaining pattern to all the clusters included in the second sequence as the evaluation value S c .

[0112]

[0113] Here, A represents the set of clusters constituting all the second segments contained in the second segment sequence. In addition, represents the set of clusters that constitute the remaining patterns. L is the set of clusters that constitute the second sequence.

[0114] In addition, S l The calculation is performed using the following formula (4).

[0115]

[0116] Here, ln is the period length of the nth period, that is, the length of the nth second segment sequence.

[0117] In addition, w in the above formula (1) s 、w c 、w lRespectively relative to S s , S c , S l The weights are all set to 1 in this embodiment.

[0118] The specifying unit 49 specifies the beginning and the end of each second segment included in the second segment sequence having the highest evaluation value among the evaluation values ​​calculated for each of the plurality of second segment sequences as the start point and the end point of each work cycle.

[0119] For example, among the evaluation values ​​calculated for the ten generated second sequence segments, the evaluation value of the second sequence segment B1 is the highest. Figure 8 As shown, cluster 0 of cluster sequence C2 corresponding to cluster 0 at the beginning of second segment B11 is determined as the starting point. In addition, cluster 4 of cluster sequence C2 corresponding to cluster 4 at the end of second segment B11 is determined as the end point. Similarly, clusters of cluster sequence C2 corresponding to clusters at the beginning and end of second segments B12 to B14 are determined as the starting point and end point, respectively.

[0120] The output unit 42 outputs the start point and the end point of the work cycle estimated by the estimation unit 41 , that is, the start point and the end point of the work cycle of the cluster sequence C2 determined by the determination unit 49 , to the storage unit 25 and stores them.

[0121] Next, refer to Figure 6 The flowchart shown in FIG. 1 illustrates a work period estimation process executed by the CPU 21A of the work period estimation device 20 .

[0122] In step S100 , the CPU 21A acquires, by reading from the storage unit 25 , a moving image captured by the camera 30 of the worker W performing a series of predetermined operations.

[0123] In step S101 , the CPU 21A calculates a motion vector sequence of the worker W based on the moving image acquired in step S100 .

[0124] In step S102 , the CPU 21A clusters the motion vector sequence calculated in step S101 , and generates a cluster sequence.

[0125] In step S103 , the CPU 21A generates a first segment sequence for each of the plurality of clusters, the first segment sequence starting with the cluster selected as the key action from the plurality of clusters clustered in step S102 , and divides the cluster sequence into a plurality of first segments.

[0126] In step S104 , the CPU 21A identifies, for each of the plurality of first segment sequences generated in step S103 , a frequent occurrence pattern of clusters common among the plurality of first segments.

[0127] In step S105 , the CPU 21A generates a second segment sequence by dividing the cluster sequence into a plurality of second segments including the frequently occurring pattern determined in step S104 .

[0128] In step S106 , the CPU 21A calculates an evaluation value regarding the similarity between the plurality of second segments for each of the plurality of second segment sequences generated in step S105 .

[0129] In step S107 , CPU 21A specifies the beginning and the end of each second segment included in the second segment sequence having the highest evaluation value among the evaluation values ​​calculated for each of the plurality of second segment sequences in step S106 as the start point and the end point of each work cycle.

[0130] In step S108 , the CPU 21A outputs the start point and the end point of the work cycle of the clustering sequence determined in step S107 to the storage unit 25 and stores them.

[0131] Thus, in this embodiment, the start and end points of the operation cycle of operator W are estimated based on the activity information related to the activities of operator W when performing a predetermined series of operations. Therefore, the start and end points of the operation cycle can be estimated without measuring the standard operation time in advance.

[0132] The above-mentioned embodiments are merely illustrative of the configuration examples of the present disclosure. The present disclosure is not limited to the above-mentioned specific forms, and various modifications can be made within the scope of the technical concept.

[0133] In addition, various processors other than the CPU may also perform the operation cycle estimation process that the CPU reads the software (program) and executes in the above-mentioned embodiments. As the processor in this case, a dedicated circuit is exemplified, which is a processor having a circuit structure specially designed for performing identification processing, such as a PLD (Programmable Logic Device) and an ASIC (Application Specific Integrated Circuit) that can change the circuit structure after manufacturing such as an FPGA (Field-Programmable Gate Array). In addition, the operation cycle estimation process may be performed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA, etc.). In addition, more specifically, the hardware structure of these various processors is a circuit that combines circuit elements such as semiconductor elements.

[0134] In addition, the disclosure of Japanese Patent Application No. 2022-116724 is incorporated in its entirety into this specification by reference. In addition, all documents, patent applications, and technical specifications described in this specification are incorporated by reference in this specification, just as if each document, patent application, and technical specification was specifically and separately described as being incorporated by reference.

Claims

1. A work cycle estimation device, comprising: An activity information acquisition unit that acquires activity information related to an activity of an operator when performing a series of predetermined operations; an estimating unit that estimates a start point and an end point of a work cycle of the operator based on the activity information; as well as The output unit outputs the estimated start point and end point of the operation cycle.

2. The operation cycle estimation device according to claim 1, wherein: The estimation unit comprises: a posture sequence generating unit that generates, based on the activity information, a posture sequence related to changes in posture based on the skeleton of the operator, wherein inherent information of the operator is removed from the posture sequence; a segment sequence generating unit that generates a segment sequence for each determined posture, the segment sequence being obtained by dividing the posture sequence into a plurality of segments so as to include the posture determined once; and The setting unit sets the beginning and the end of each segment included in a segment sequence having the highest degree of similarity among the plurality of segment sequences generated for each of the specified postures as the start point and the end point of each work cycle.

3. The operation cycle estimation device according to claim 2, wherein: The posture sequence generating unit comprises: a motion vector sequence calculation unit that calculates a motion vector sequence of the operator based on the activity information; and a clustering unit, clustering the motion vector sequence and generating a cluster sequence, The segment sequence generating unit comprises: A first segment sequence generating unit, for each of a plurality of clusters, generates a first segment sequence, wherein the first segment sequence starts with a cluster selected from the plurality of clusters and divides the cluster sequence into a plurality of first segments; a frequent occurrence pattern determination unit that determines, for each of the plurality of first segment sequences generated, a frequent occurrence pattern of clusters common among the plurality of first segments; and A second segment sequence generating unit generates a second segment sequence, wherein the second segment sequence divides the cluster sequence into a plurality of second segments including the frequently occurring pattern. The setting unit includes: an evaluation value calculation unit that calculates, for each of the plurality of generated second segment sequences, an evaluation value related to a degree of similarity between the plurality of second segments; and The specifying unit specifies the beginning and the end of each of the second segments included in the second segment sequence having the highest evaluation value among the evaluation values ​​calculated for each of the plurality of second segment sequences as the start point and the end point of each operation cycle.

4. The operation cycle estimation device according to claim 3, wherein: The clustering unit removes from the cluster sequence a cluster that exists only near the beginning or the end of the cluster sequence, that is, a cluster whose ratio to the length of the cluster sequence is less than or equal to a predetermined first threshold.

5. The operation cycle estimation device according to claim 3, wherein: The second segment sequence generating unit uses half of the cycle length of the plurality of second segments included in the second segment sequence as a second threshold value, and combines a second segment having a cycle length equal to or less than the second threshold value with a previous second segment.

6. The operation cycle estimation device according to claim 3, wherein: The evaluation value calculation unit excludes second segments, for which the similarities between a plurality of second segments are all equal to or less than a third threshold value determined in advance, from calculation targets of the evaluation value.

7. The operation cycle estimation device according to claim 3, wherein: The first segment sequence generating unit generates an end-excluded segment sequence excluding the first segment at the end and a start-excluded segment sequence excluding the first segment at the beginning. The frequent appearance pattern determination unit determines a frequent appearance pattern obtained by combining a frequent appearance pattern of the end-excluded segment sequence and a frequent appearance pattern of the beginning-excluded segment sequence as a frequent appearance pattern of a cluster common to a plurality of the first segments.

8. The operation cycle estimation device according to claim 3, wherein: The second segment sequence generation unit generates: a second segment sequence obtained by dividing the cluster sequence into a second segment in which a pattern other than the second segment candidate including the frequently occurring pattern is added to the beginning of the second segment candidate adjacent to the pattern, and a second segment sequence obtained by dividing the cluster sequence into a second segment in which a pattern other than the second segment candidate is added to the end of the second segment candidate adjacent to the pattern.

9. The operation cycle estimation device according to claim 2, wherein: The posture sequence is a sequence based on the postures of the skeletons of the upper body and the lower body of the worker.

10. The operation cycle estimation device according to claim 2, wherein: The pose sequence generation unit generates a pose sequence from which the influence of a camera viewpoint is removed.

11. The operation period estimation device according to any one of claims 2 to 10, wherein: The posture sequence generation unit generates a skeleton sequence of the operator based on the movement information, and performs at least one of an interpolation process and a smoothing process on the generated skeleton sequence to generate the posture sequence.

12. A method for estimating a work cycle, wherein: The computer performs the following processing: Acquiring activity information related to the activities of an operator when performing a predetermined series of operations; estimating a start point and an end point of a work cycle of the operator based on the activity information; as well as Output the estimated start and end points of the work cycle.

13. A cycle time estimation program, wherein: The operation cycle estimation program causes the computer to execute the following processing: Acquiring activity information related to the activities of an operator when performing a predetermined series of operations; estimating a start point and an end point of a work cycle of the operator based on the activity information; as well as Output the estimated start and end points of the work cycle.

Images