Information processing device and information processing program

Abstract

To provide an information processing apparatus that extracts reference time-series information from pieces of time-series information.SOLUTION: An information processing apparatus according to an embodiment has an interface that is connected to a sensor that acquires moving images obtained by photographing work, and a processor that acquires an image through the interface and performs arithmetic processing. The processor collects time-series information including the feature quantity of the work from the moving images, and applies clustering to the time-series information to classify the time-series information into a first class. Of the first class, for a second class including three or more pieces of time-series information, the processor calculates at least one of the distance and the similarity between each time-series information in the second class and a comparison object, for each time-series information in the second class, calculates statistics from at least one of the distance and the similarity, and on the basis of the statistics of each time-series information in the second class, extracts, from the pieces of time-series information in the second class, reference time-series information to which division information indicating the division point of the work should be applied.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an information processing apparatus and an information processing program. [Background Art]

[0002] Information processing apparatuses that measure work time in a production line are known. As one such information processing apparatus, there is one that uses a method of measuring work time by collecting time-series information including feature amounts of work, associating reference time-series information provided with division information indicating work division points with the time-series information, and measuring the work time of each of a plurality of works obtained by dividing the work at the division points based on the time-series information corresponding to the reference time-series information. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2020-119198 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] In the above method, reference time-series information provided with division information indicating work division points is required in order to measure work time. Which piece of time-series information is provided with division information to create the reference time-series information affects the measurement accuracy of work time. Therefore, it is important to extract appropriate reference time-series information to be provided with division information from collected time-series information. However, the above method is premised on that reference time-series information is provided in advance, and cannot extract reference time-series information from collected time-series information.

[0005] The problem to be solved by the present invention is to provide an information processing apparatus and an information processing program that extract reference time-series information from collected time-series information. [Means for Solving the Problem]

[0006] The information processing device according to this embodiment includes an interface connected to a sensor that acquires video footage of the work, and a processor that acquires the video via the interface and performs calculations. The processor processes the video, The feature vectors extracted from each frame of the video were arranged in the time direction. The processor collects time-series information and applies clustering to it to classify the time-series information into a first class. For the second class of the first class that contains three or more time-series information, the processor processes each time-series information within the second class. , or reference time series information calculated from other time series information within the second class or from time series information within the second class. At least one of the distance and similarity to the comparison target Using DTW (Dynamic Time Warping) Calculate and, for each time series information in the second class, from at least one of distance and similarity It is one of the following: mean, median, minimum, or maximum. Calculate statistics, and based on the statistics of each time series information within the second class, extract reference time series information from the time series information within the second class to which segmentation information indicating the division points of the work should be added. The work time is measured by adding division information to the reference time series information, and then dividing the work at the division points based on the time series information associated with the reference time series information to which the division information has been added, and measuring the work time for each of the multiple tasks obtained. do. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a diagram showing the hardware configuration of an information processing device according to an embodiment. [Figure 2] Figure 2 is a flowchart showing a first example of operation of the information processing device. [Figure 3] Figure 3 schematically illustrates how reference time-series information is extracted when there is only one time-series information item in the first example of operation of the information processing device. [Figure 4] Figure 4 schematically shows how reference time-series information is extracted when there are two time-series pieces of information in the first example of operation of the information processing device. [Figure 5] Figure 5 schematically shows how reference time-series information is extracted when there are three or more time-series pieces of information in the first example of operation of the information processing device. [Figure 6] Figure 6 is a flowchart showing the details of the calculation of distance, similarity, and statistics in the first example of operation. [Figure 7] Figure 7 is a flowchart showing the details of extracting and saving reference time series information based on statistics in the first operational example. [Figure 8] Figure 8 is a flowchart showing a second example of the operation of the information processing device. [Figure 9] Figure 9 schematically shows how reference time-series information is extracted in the second example of operation of the information processing device. [Figure 10] Figure 10 is a flowchart showing a third example of operation of the information processing device. [Figure 11] Figure 11 schematically shows how reference time-series information is extracted in the third example of operation of the information processing device. [Figure 12] Figure 12 is a flowchart showing a fourth example of operation of the information processing device. [Figure 13] Figure 13 schematically illustrates how reference time-series information is extracted for each class in the fourth example of operation of the information processing device. [Figure 14A] Figure 14A is a flowchart showing a part of the fifth operation example of the information processing device. [Figure 14B] Figure 14B is a flowchart showing a portion of the fifth operational example of the information processing device. [Figure 15] Figure 15 schematically shows how reference time-series information is extracted for each class in the fifth example of operation of the information processing device. [Figure 16A] Figure 16A is a flowchart showing a portion of the sixth operational example of the information processing device. [Figure 16B] Figure 16B is a flowchart showing a portion of the sixth operational example of the information processing device. [Figure 17] Figure 17 is a flowchart showing the seventh example of operation of the information processing device. [Modes for carrying out the invention]

[0008] An information processing apparatus according to an embodiment will be described below. The information processing apparatus according to the embodiment is an apparatus that collects time-series information including feature quantities of a work from a moving image obtained by capturing the work, and extracts reference time-series information to which division information indicating a division point of the work should be assigned from the collected time-series information.

[0009] (Hardware Configuration) First, the hardware configuration of the information processing apparatus 10 according to the embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the hardware configuration of the information processing apparatus 10 according to the embodiment. The information processing apparatus 10 is constituted by, for example, a computer. Examples of the computer include a personal computer, a server computer, and the like. The computer executes each function of the information processing apparatus 10 by running an information processing program.

[0010] As shown in FIG. 1, the information processing apparatus 10 includes a processor 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input device 15, an output device 16, a sensor interface 17, and a bus 18. In the drawings, "interface" is abbreviated as "I / F".

[0011] The processor 11, the ROM 12, the RAM 13, the storage 14, the input device 15, the output device 16, and the sensor interface 17 are electrically connected to each other via the bus 18, and can transmit and receive data and commands via the bus 18.

[0012] The processor 11 is hardware that executes programs and processes data. The processor 11 may include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), and the like. The processor 11 controls the entirety of the ROM 12, the RAM 13, the storage 14, the input device 15, the output device 16, and the sensor interface 17.

[0013] ROM12 is a non-volatile memory that constitutes part of the main memory. ROM12 non-temporarily stores the startup program necessary for starting the information processing device 10. The processor 11 starts the information processing device 10 by executing the program in ROM12. ROM12 is, for example, composed of EPROM (Erasable Programmable Read Only Memory) and stores various startup settings in addition to the startup program.

[0014] RAM13 is a volatile memory that constitutes part of the main memory. RAM13 temporarily stores the program necessary for processing by the processor 11 and the data necessary for executing the program. By executing the program in RAM13, the processor 11 performs calculations on the data in RAM13 and stores the calculation results in RAM13.

[0015] Storage 14 is a non-volatile memory that constitutes auxiliary storage. Storage 14 is composed of, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Storage 14 non-temporarily stores programs executed by the processor 11 and the data necessary for program execution. The processor 11 reads the programs and data from storage 14 into RAM 13 and executes various functions by executing the programs.

[0016] The input device 15 may include a keyboard, mouse, etc. However, it is not limited to these, and may include any other input device. The output device 16 may include a display, etc. However, it is not limited to these, and may include any other output device. The input device 15 and the output device 16 may be configured as an input / output device having the functions of both.

[0017] The sensor interface 17 enables connection to the sensor 20 that acquires video. For example, the sensor interface 17 may be a wired interface or a wireless interface. A wired interface includes a port to which the device is connected, etc. A wireless interface includes Bluetooth®, WiFi®, etc.

[0018] Sensor 20 includes, for example, a camera or a network camera. The video is, for example, a video of work being done on a production line.

[0019] When the computer starts up, the processor 11 executes a program in the ROM 12 and loads the OS into the RAM 13 to start up. Under the control of the OS, the processor 11 monitors instruction inputs and the connection of external devices. Also, under the control of the OS, the processor 11 sets up a program area and a data area in the RAM 13. In response to an instruction input to start an information processing program, the processor 11 loads the information processing program from the storage 14 into the program area of ​​the RAM 13, and loads the data necessary for the execution of the information processing program from the storage 14 into the data area of ​​the RAM 13. The processor 11 performs calculations on the data in the data area according to the information processing program and writes the calculation results to the data area. Through these operations, the processor 11, RAM 13, storage 14, input device 15, output device 16, and sensor interface 17 work together to execute each function of the information processing device 10.

[0020] (Example of operation 1) Next, with reference to Figure 2, a first example of operation of the information processing device 10 will be described. Figure 2 is a flowchart of the first example of operation of the information processing device 10. Each operation of the information processing device 10 in the flowchart of Figure 2 is performed by the processor 11 executing an information processing program. In other words, the information processing program is a program that causes the processor 11 to execute each operation of the information processing device 10 in the flowchart of Figure 2. In the following description, the process will begin with the start of the execution of the information processing program by the processor 11. It will also be assumed that the sensor 20, such as a camera, transmits video footage of the work on the production line to the information processing device 10.

[0021] First, in ACT11, the information processing device 10 collects time-series information containing work features from video transmitted from sensors 20 such as cameras. For example, one piece of time-series information is acquired as follows.

[0022] First, video footage of one cycle of work is acquired. This video footage is obtained by detecting specific postures that indicate the start and end of the work, and extracting the footage from the video during that period. Feature extraction (human action recognition, skeletal detection, object detection, and vectorization of the image itself) is applied to the video footage of one cycle to extract feature vectors for each frame, and by arranging these in the chronological direction, a single time-series piece of information is obtained. The information processing device 10 stores the acquired time-series information in the storage 14.

[0023] In parallel with collecting time-series information, the information processing device 10 awaits notification to start extracting reference time-series information to which division information indicating the division points of the work should be added. The notification to start extracting reference time-series information is sent from the input device 15 to the processor 11 in response to the input of a command to start extracting reference time-series information to the input device 15.

[0024] The information processing device 10 continues to collect time-series information in ACT11 as long as it does not receive a notification to start extracting reference time-series information (while "No" is displayed in ACT12).

[0025] When the information processing device 10 receives a notification to start extracting reference time-series information (if the answer in ACT12 is Yes), it determines in ACT13 whether it has detected the completion of collecting three or more time-series information. Detecting the completion of collecting three or more time-series information means that three or more time-series information items have been collected. There is no upper limit to the number of time-series information items.

[0026] If the information processing device 10 does not detect the completion of collection of three or more time-series information (in the case of No in ACT13), it determines in ACT14 whether it has detected the completion of collection of two time-series information. Detecting the completion of collection of two time-series information means that two time-series information items have been collected.

[0027] If the information processing device 10 does not detect the completion of collection of two time-series information (in the case of No in ACT14), it determines in ACT15 whether it has detected the completion of collection of one time-series information. Detecting the completion of collection of one time-series information means that one time-series information has been collected.

[0028] If the information processing device 10 does not detect the completion of collecting one time-series piece of information (in the case of No in ACT15), it returns to the time-series piece of information collection operation in ACT11. Not detecting the completion of collecting one time-series piece of information means that zero time-series pieces of information have been collected, that is, no time-series pieces of information have been collected.

[0029] When the information processing device 10 detects that the collection of one time-series piece of information has been completed (if the answer is Yes in ACT 15), it saves that one time-series piece of information as reference time-series information in the storage 14 in ACT 16.

[0030] Figure 3 schematically illustrates how reference time series information is extracted when only one time series information is collected. In this case, there is only one time series information TSI, and no other time series information TSIs exist, so that single time series information TSI is extracted as the reference time series information TSIref.

[0031] Subsequently, the information processing device 10 determines in ACT21 whether it has received a notification of the termination of the information processing program. The notification of the termination of the information processing program is transmitted from the input device 15 to the processor 11 in response to the input of a termination instruction for the information processing program to the input device 15.

[0032] If the information processing device 10 has not received a notification that the information processing program has finished (in the case of No in ACT21), it returns to the operation of collecting time-series information in ACT11.

[0033] When the information processing device 10 receives a notification that the information processing program has finished (in the case of Yes in ACT21), it terminates the operation of extracting reference time-series information.

[0034] When the information processing device 10 detects that the collection of two time-series information has been completed (in the case of Yes in ACT 14), in ACT 17, it randomly selects one of the two time-series information as the reference time-series information and stores it in the storage 14.

[0035] Subsequently, the information processing device 10 performs the operation of determining whether it has received the termination notification of the information processing program in ACT21. The operation in ACT21 is as described above.

[0036] Figure 4 schematically illustrates how reference time series information is extracted when two time series data points are collected. In this case, since there are two time series data points (TSI), one of the two TSI data points is randomly selected as the reference time series data point (TSIref). The reason for randomly selecting one of the two TSI data points as the reference time series data point (TSIref) is as follows.

[0037] As will be described later, if there are three or more time series information TSIs, the distance and similarity of each time series information with the comparison target are calculated, a statistic is calculated for each time series information from at least one of the distance and similarity, and the reference time series information is extracted based on the statistic for each time series information.

[0038] In contrast, when two time series data points are collected, at least one of the distance and similarity metric is calculated for only one set of data points. Therefore, the statistics calculated for both sets of time series data points are identical, and there is no difference between them. For this reason, one of the two time series data points (TSI) is randomly selected as the reference time series data point (TSIref).

[0039] For the purposes of the following explanation, distance and similarity will be referred to as "distance" or "similarity" for convenience. In other words, "distance" or "similarity" means distance, or similarity, or both.

[0040] If the information processing device 10 detects that the collection of three or more time-series information has been completed (in the case of Yes in ACT13), in ACT18, it calculates the distance and similarity of each time-series information to the comparison target using DTW (Dynamic Time Warping) or the like.

[0041] Next, in ACT19, the information processing device 10 calculates statistics for each time series of information from distance and similarity. These statistics include, for example, the mean, median, minimum, and maximum values.

[0042] Next, the information processing device 10 extracts reference time series information from the time series information in ACT20 based on the statistical values ​​of each time series information and stores it in the storage 14.

[0043] Subsequently, the information processing device 10 performs the operation of determining whether it has received the termination notification of the information processing program in ACT21. The operation in ACT21 is as described above.

[0044] Figure 5 schematically illustrates how reference time series information is extracted when three or more time series data points are collected. The left side of Figure 5 shows the collected time series data TSIs and examples of their distribution. As an example, Figure 5 depicts a case where four time series data TSIs have been collected. The center of Figure 5 schematically illustrates how the distance and similarity between each time series data TSI and the comparison target are calculated. The right side of Figure 5 schematically illustrates how one of the four time series data TSIs is extracted as reference time series data TSIref. Here, the time series data TSI located approximately in the center of the distribution is extracted as reference time series data TSIref.

[0045] As described above, the information processing device and information processing program according to the embodiment make it possible to extract reference time-series information from the collected time-series information.

[0046] (Calculation of distance and similarity, and calculation of statistics) Next, referring to Figure 6, we will explain the details of the calculation of distance and similarity in ACT18 and the calculation of statistics in ACT19. Figure 6 is a flowchart showing the details of the calculation of distance and similarity in ACT18 and the calculation of statistics in ACT19.

[0047] In ACT31, the information processing device 10 selects one comparison target S1. The comparison target S1 may be one of the collected time-series information or other information. The number of comparison targets S1 may be one or multiple.

[0048] Next, the information processing device 10 selects one time-series information S2 in ACT32. Here, the requirement that the comparison target S1 and the time-series information S2 are different from each other must be met. That is, if the comparison target S1 is one of the time-series information, then it is necessary not to select the same time-series information as the time-series information S2. If the comparison target S1 is other information, then the above requirement is automatically met.

[0049] Next, in ACT33, the information processing device 10 calculates the distance and similarity between the comparison target S1 and the time series information S2 using DTW (Dynamic Time Warping) or the like.

[0050] Next, the information processing device 10 determines in ACT34 whether all time-series information has been selected as time-series information S2.

[0051] If the determination results in not all time-series information being selected as time-series information S2 (i.e., the result is No in ACT34), the information processing device 10 returns to the time-series information S2 selection operation in ACT32.

[0052] If, as a result of the determination, all time series information is selected as time series information S2 (Yes in ACT34), then in ACT35, the information processing device 10 calculates statistics for each time series information from the distance and similarity calculated for all time series information in ACT32 to ACT34.

[0053] Subsequently, the information processing device 10 determines in ACT36 whether all comparison targets have been selected as comparison target S1.

[0054] If the determination results in not all comparison targets being selected as comparison target S1 (i.e., the result is No in ACT36), the information processing device 10 returns to the operation of selecting comparison target S1 in ACT31.

[0055] If, as a result of the determination, all comparison targets are selected as comparison target S1 (i.e., Yes in ACT36), the information processing device 10 terminates the calculation of distance and similarity and the calculation of statistics.

[0056] (Extraction and storage of reference time series information based on statistical data) Next, with reference to Figure 7, we will explain the details of extracting and storing reference time series information based on statistics in ACT20. Figure 7 is a flowchart showing the details of extracting and storing reference time series information based on statistics in ACT20.

[0057] The information processing device 10 identifies the minimum or maximum value among the statistics in ACT37.

[0058] Next, the information processing device 10 extracts the time series information with the minimum or maximum value in ACT38 as the reference time series information.

[0059] Next, in ACT39, the information processing device 10 stores the reference time-series information extracted in ACT38 in the storage 14.

[0060] (Second example of operation) Next, with reference to Figure 8, a second example of operation of the information processing device 10 will be described. Figure 8 is a flowchart of the second example of operation of the information processing device 10. In this second example of operation of the information processing device 10, the comparison target is each of the collected time-series pieces of information.

[0061] In comparison with the flowchart in Figure 2, the operations of ACT11-ACT17 and ACT19-ACT21 are the same in the flowchart in Figure 8, and their operations are as described above. In other words, the second example of operation is an example in which the operation of ACT22 is performed instead of the operation of ACT18 in the first example of operation. Below, we will explain the differences from the first example of operation, namely the operation of ACT22.

[0062] In the second example of operation, if the information processing device 10 detects that the collection of three or more time-series information has been completed (Yes in ACT13), in ACT22, it calculates the distance and similarity between each time-series information using DTW (Dynamic Time Warping) or the like. That is, it calculates the distance and similarity between any one of the collected time-series information and the other time-series information. The details of calculating the distance and similarity between each time-series information are the same as when one of the collected time-series information is selected as the comparison target S1 in the flowchart of Figure 6.

[0063] In the second operational example, Figure 9 schematically illustrates how reference time series information is extracted when three or more time series information items are collected. The left side of Figure 9 shows the collected time series information TSI and an example of its distribution. As an example, Figure 9 depicts a case where four time series information TSI items have been collected.

[0064] The second figure from the left in Figure 9 schematically shows how the distance and similarity between each time series information TSI are calculated. The numbers written between the time series information TSIs represent the distance. In other words, in the second example of operation, the information processing device 10 calculates the distance between each time series information in ACT22.

[0065] The third figure from the left in Figure 9 schematically shows how the average distance between each time series information TSI is calculated as a statistical quantity for each time series information TSI. The numbers written for each time series information TSI represent the average value, which is a statistical quantity for each time series information TSI. In other words, in the second example of operation, the information processing device 10 calculates the average distance between each time series information TSI as a statistical quantity in ACT19 for each time series information TSI.

[0066] The fourth image from the left in Figure 9, i.e., on the right, schematically shows how one of the four time series information TSIs is extracted as the reference time series information TSIref based on the average value of the distances between each time series information TSI. Here, the time series information TSI with the smallest average value of the distances between each time series information TSI is extracted as the reference time series information TSIref. In other words, in the second example of operation, the information processing device 10 extracts the time series information TSI with the smallest average value of the distances between each time series information TSI as the reference time series information TSIref in ACT20 and stores it in storage 14.

[0067] According to the second example of operation, it becomes possible to extract reference time-series information while reducing the amount of memory (RAM 13 and storage 14) used.

[0068] (Third example of operation) Next, with reference to Figure 10, a third example of operation of the information processing device 10 will be described. Figure 10 is a flowchart of the third example of operation of the information processing device 10. The third example of operation of the information processing device 10 is an example in which the comparison target is reference time-series information.

[0069] In comparison with the flowchart in Figure 2, the operations of ACT11-ACT17 and ACT19-ACT21 are the same in the flowchart in Figure 10, and their operations are as described above. In other words, the third example of operation is one in which the operations of ACT23 and ACT24 are performed instead of the operation of ACT18 in the first example of operation. Below, we will focus on explaining the differences from the first example of operation, namely the operations of ACT23 and ACT24.

[0070] In the third example of operation, if the information processing device 10 detects that the collection of three or more time-series information items has been completed (Yes in ACT13), it calculates reference time-series information from the collected time-series information in ACT23. The reference time-series information can be anything that can be calculated from the collected time-series information. The number of reference time-series information items can be one or more. The reference time-series information is, for example, average time-series information. The average time-series information is, for example, information that is located at the centroid of the distribution of the collected time-series information and is obtained by averaging the collected time-series information.

[0071] Next, the information processing device 10 calculates the distance and similarity between each time series information and the reference time series information in ACT24 using DTW (Dynamic Time Warping), etc. The details of calculating the distance and similarity between each time series information and the reference time series information are the same as when the reference time series information of the collected time series information is selected as the comparison target S1 in the flowchart of Figure 6.

[0072] In the third operational example, Figure 11 schematically illustrates how, when three or more time series data points are collected, the average time series data is calculated and the reference time series data is extracted. The left side of Figure 11 shows the collected time series data TSI and an example of its distribution. As an example, Figure 11 depicts a case where four time series data TSIs have been collected.

[0073] The second image from the left in Figure 11 schematically shows how the average time series information ATSI of the collected time series information is calculated. In other words, in the third example of operation, the information processing device 10 calculates the average time series information ATSI of the collected time series information TSI in ACT23.

[0074] The third figure from the left in Figure 11 schematically shows how the distance and similarity between each time series information (TSI) and the mean time series information (ATSI) are calculated. The numerical values ​​between each time series information (TSI) and the mean time series information (ATSI) represent the distance. In other words, in the third example of operation, the information processing device 10 calculates the distance between each time series information (TSI) and the mean time series information (ATSI) in ACT24.

[0075] The fourth image from the left in Figure 11, i.e., on the right, schematically shows how one of the four time series information TSIs is extracted as the reference time series information TSIref based on the distance between each time series information TSI and the mean time series information ATSI. Here, the time series information TSI with the smallest distance between each time series information TSI and the mean time series information ATSI is extracted as the reference time series information TSIref. That is, in the third example of operation, the information processing device 10 extracts the time series information TSI with the smallest average distance between each time series information TSI and the mean time series information ATSI as the reference time series information TSIref in ACT19 and ACT20 and stores it in storage 14.

[0076] In the third example of operation, the statistic used is the distance between each time series information TSI and the mean time series information ATSI. In other words, the statistic calculated in ACT19 is the distance between each time series information TSI and the mean time series information ATSI.

[0077] According to the third example of operation, it becomes possible to extract reference time series information while reducing the computational load.

[0078] (Fourth example of operation) Next, with reference to Figure 12, a fourth example of operation of the information processing device 10 will be described. Figure 12 is a flowchart of the fourth example of operation of the information processing device 10. Each operation of the information processing device 10 in the flowchart of Figure 12 is performed by the processor 11 executing an information processing program. In other words, the information processing program is a program that causes the processor 11 to execute each operation of the information processing device 10 in the flowchart of Figure 12. The prerequisites for operation are the same as in the first example of operation.

[0079] First, in ACT41, the information processing device 10 collects time-series information from the video transmitted from the sensor 20, such as a camera. The method of collecting time-series information is the same as in the first example of operation.

[0080] The information processing device 10 waits to receive a notification to start extracting reference time-series information, in parallel with collecting time-series information.

[0081] The information processing device 10 continues to collect time-series information in ACT41 as long as it does not receive a notification to start extracting reference time-series information (while "No" is displayed in ACT42).

[0082] When the information processing device 10 receives a notification to start extracting reference time series information (if the answer is Yes in ACT42), in ACT43, it applies clustering to the collected time series information to classify the time series information into classes. As a clustering method, for example, the K-means method or the X-means method can be used. Other clustering methods may also be used.

[0083] The number of classes must be one or more. In other words, the number of classes can be one or more. If there are multiple classes, the multiple classes may correspond to, for example, different tasks, tasks performed by different workers, or tasks performed by the same worker but with different skill levels.

[0084] Next, in ACT44, the information processing device 10 determines whether it has detected the completion of collecting three or more time-series information items within each class. Detecting the completion of collecting three or more time-series information items within a class means that there are three or more time-series information items within that class. There is no upper limit on the number of time-series information items.

[0085] If the information processing device 10 does not detect the completion of collection of three or more time-series information items within a class (in the case of No in ACT44), then in ACT45, it determines for each class whether it has detected the completion of collection of two time-series information items within that class. Detecting the completion of collection of two time-series information items within a class means that there are two time-series information items within that class.

[0086] If the information processing device 10 does not detect the completion of collection of two time-series information items within a class (in the case of No in ACT45), then in ACT46, it determines whether it has detected the completion of collection of one time-series information item within each class. Detecting the completion of collection of one time-series information item within a class means that there is one time-series information item within the class.

[0087] If the information processing device 10 does not detect the completion of collecting one time-series piece of information within a class (in the case of No in ACT46), it returns to the time-series piece of information collection operation in ACT41. For each class, not detecting the completion of collecting one time-series piece of information within the class means that zero time-series pieces of information have been collected, that is, no time-series pieces of information have been collected.

[0088] If the information processing device 10 detects that the collection of one time-series piece of information within a class has been completed (in the case of Yes in ACT46), then in ACT47, it saves that one time-series piece of information for that class as reference time-series information in the storage 14.

[0089] Subsequently, the information processing device 10 determines in ACT52 whether it has received a notification that the information processing program has ended.

[0090] If the information processing device 10 has not received a termination notification for the information processing program (in the case of No in ACT52), it returns to the operation of collecting time-series information in ACT41.

[0091] When the information processing device 10 receives a notification that the information processing program has finished (in the case of Yes in ACT52), it terminates the operation of extracting reference time-series information.

[0092] If the information processing device 10 detects that the collection of two time-series information items within a class has been completed (in the case of Yes in ACT45), in ACT48, it randomly selects one of those two time-series information items for that class as the reference time-series information item and stores it in the storage 14.

[0093] Subsequently, the information processing device 10 performs the operation of determining whether it has received the termination notification of the information processing program in ACT52. The operation in ACT52 is as described above.

[0094] If the information processing device 10 detects that the collection of three or more time-series information items within a class has been completed (in the case of Yes in ACT44), in ACT49, it calculates the distance and similarity of each time-series information item with a comparison target for each class using DTW (Dynamic Time Warping) or the like. The comparison target is, for example, each time-series information item or a reference time-series information item. However, the comparison target is not limited to these and may be other information.

[0095] Next, in ACT50, the information processing device 10 calculates statistics for each time series of information for each class, based on distance and similarity. These statistics include, for example, the mean, median, minimum, and maximum values.

[0096] Next, in ACT51, the information processing device 10 extracts reference time series information from the time series information based on the statistical values ​​of each time series information for each class and stores it in the storage 14.

[0097] Subsequently, the information processing device 10 performs the operation of determining whether it has received the termination notification of the information processing program in ACT52. The operation in ACT52 is as described above.

[0098] Figure 13 schematically illustrates how clustering is applied to the collected time-series information, resulting in its classification into classes, and how reference time-series information is extracted for each class.

[0099] The left side of Figure 13 shows the collected time-series information TSI and examples of their distribution. In Figure 13, three representative time-series information TSIs are labeled.

[0100] The center of Figure 13 shows the results of applying clustering to the time-series information TSI on the left side of Figure 13. Here, as an example, the time-series information TSI on the left side of Figure 13 is shown to have been classified into three classes (Class A, Class B, and Class C).

[0101] The right side of Figure 13 shows how one reference time series information TSIref has been extracted for each of Class A, Class B, and Class C. The method for extracting one reference time series information TSIref in each class can be any of the first to third operational examples described above. For example, as explained in the second operational example, the distance between each time series information TSI within each class can be calculated, and the time series information TSI with the smallest distance can be extracted as the reference time series information TSIref.

[0102] For example, the method for extracting the reference time series information TSIref is the same for all three classes. However, the method for extracting the reference time series information TSIref may differ for each class, without being limited to this.

[0103] According to the fourth example of operation, because time-series information is classified into classes, it becomes possible to extract reference time-series information for each different operation pattern.

[0104] (Reference time-series information used to measure work time) Here, we will explain the reference time series information used for measuring work time. In the fourth example of operation, one reference time series information is extracted for each class. Therefore, for example, multiple reference time series information are extracted overall. In that case, we will add a note on how multiple reference time series information are used in measuring work time. Note that, as preparation for measuring work time, segmentation information is attached to the reference time series information.

[0105] The measurement of work time is performed using the reference time series information in response to newly collected time series information that is different from the time series information used to extract the reference time series information.

[0106] In one example, the distance and similarity between each of the newly collected input time-series data points and each of the multiple reference time-series data points are calculated, and the reference time-series data point with the smallest distance and similarity is used to measure the work time. In other words, one of the multiple reference time-series data points is selected and used to measure the work time.

[0107] In another example, clustering is applied to newly collected time-series input data to classify it into categories, and for each class of input time-series data, work time is measured using reference time-series data of the same class. For example, for time-series input data of class A, work time is measured using reference time-series data of class A.

[0108] Furthermore, for time series information input from a different class than the reference time series information, it is possible to select one of several reference time series information using the method described above and use it to measure the work time.

[0109] (Fifth example of operation) Next, a fifth example of operation of the information processing device 10 will be described with reference to Figures 14A and 14B. Figures 14A and 14B are flowcharts of the fifth example of operation of the information processing device 10.

[0110] In comparison with the flowchart in Figure 12, the flowcharts in Figures 14A and 14B show that the operations of ACT41-ACT42 and ACT44-ACT52 are the same, and their operations are as described above. In other words, the fifth example of operation is an example in which the operations of ACT53-ACT56 are performed instead of the operation of ACT43 in the fourth example of operation. Below, we will focus on explaining the differences from the fourth example of operation, namely the operations of ACT53-ACT56.

[0111] In the fifth operational example, when the information processing device 10 receives a notification to start extracting reference time-series information (if the answer is Yes in ACT42), in ACT53, it applies clustering to the collected time-series information to classify the time-series information into classes and obtains the current clustering result. The information processing device 10 also saves the current clustering result as a log to the storage 14.

[0112] Next, the information processing device 10 refers to the logs stored in the storage 14 in ACT54 and obtains past clustering results.

[0113] Furthermore, when the information processing device 10 obtains the current clustering result in ACT53, it may obtain past clustering results and obtain the current clustering result by referring to the past clustering results.

[0114] Next, the information processing device 10 calculates the current clustering result and the change in past clustering results in ACT55. For example, the change in clustering results is the increase or decrease in the number of classes and the amount of movement of the class centroids.

[0115] Next, the information processing device 10 determines whether it has detected a change in the clustering result that exceeds a threshold in ACT56.

[0116] If the determination does not detect a change in the clustering result that exceeds the threshold (i.e., No in ACT56), the information processing device 10 performs the operation to determine receipt of the termination notification of the information processing program in ACT52. The operation in ACT52 is as described in the fourth operation example.

[0117] If the determination detects a change in the clustering result that exceeds a threshold (i.e., Yes in ACT56), the information processing device 10 performs the operation of ACT44. The operations from ACT44 onward are as described in the fourth operation example.

[0118] In the fifth operational example, Figure 15 schematically illustrates how reference time series information is extracted for each class. In Figure 15, the upper section shows the past clustering, and the lower section shows the current clustering. The left side shows the collected time series information TSI and its distribution examples, the center shows the clustering results, and the right side shows how the reference time series information TSIref has been extracted for each class. The meaning of Figure 15 is the same as that of Figure 13.

[0119] Comparing the results of past clustering with the current clustering, the current clustering generates a new class D in addition to classes A, B, and C. In other words, the number of classes has increased from 3 to 4. This corresponds to detecting a change in the clustering results that exceeds a threshold. Therefore, as shown in the lower right of Figure 15, one new reference time series information TSIref is extracted for each class. The method for extracting one reference time series information TSIref for each class is as explained in the fourth example of operation.

[0120] According to the fifth example of operation, since past clustering results are referenced, it becomes possible to continuously extract reference time series information.

[0121] (Other methods for extracting reference time series information) Alternatively, instead of extracting a new reference time series information TSIref for each class, other methods for extracting reference time series information TSIrefs can also be considered. This will be explained below.

[0122] Classes A, B, and C are common to both past and present clustering results. Of course, the time series information TSIs included in classes A, B, and C differ between past and present clustering results. However, the time series information TSIs classified in the same class are similar, and the reference time series information TSIref extracted from these time series information TSIs is expected to be similar between past and present clustering.

[0123] For example, the TSIref reference time series information for Class A extracted in the current clustering is expected to be similar to the TSIref reference time series information for Class A extracted in past clustering.

[0124] Based on this reasoning, for classes A, B, and C, the current clustering process uses the same reference time series information TSIref from past clustering processes without extracting new reference time series information TSIref. Only for class D is new reference time series information TSIref extracted.

[0125] This method significantly reduces the processing load required to extract the reference time series information TSIref.

[0126] (Sixth example of operation) Next, a sixth example of operation of the information processing device 10 will be described with reference to Figures 16A and 16B. Figures 16A and 16B are flowcharts of the sixth example of operation of the information processing device 10.

[0127] In comparison with the flowchart in Figure 2, the flowcharts in Figures 16A and 16B show that the operations of ACT11-ACT18 and ACT21 are the same, and their operations are as described above. In other words, the sixth example of operation is an example in which the operations of ACT61-ACT68 are performed instead of the operations of ACT19-ACT20 in the first example of operation. Below, we will focus on explaining the differences from the first example of operation, namely the operations of ACT61-ACT68.

[0128] In the sixth example of operation, after the operation of ACT18, the information processing device 10 removes distances and similarities that are smaller than the first threshold and larger than the second threshold in ACT61. Here, the second threshold is larger than the first threshold. Simply put, it removes distances and similarities that deviate significantly from the mean, for example. Hereafter, such values ​​that deviate significantly from the mean will be referred to as "outliers," and the removal of outliers will be referred to as "outlier removal." The information processing device 10 also removes time-series information corresponding to outliers when removing distances and similarities of outliers. This is also referred to as "outlier removal."

[0129] Next, the information processing device 10 determines whether it has detected in ACT62 that there are three or more time-series information items after outlier removal.

[0130] If the information processing device 10 does not detect that there are three or more time-series information items after outlier removal (in the case of No in ACT62), it determines in ACT63 whether it has detected that there are two time-series information items after outlier removal.

[0131] If the information processing device 10 does not detect that there are two time-series data points after outlier removal (in the case of No in ACT63), it determines in ACT64 whether it has detected that there is one time-series data point after outlier removal.

[0132] If the information processing device 10 does not detect that there is one time-series information after outlier removal (in the case of No in ACT64), it returns to the operation of collecting time-series information in ACT11.

[0133] If the information processing device 10 detects that there is one time series information after outlier removal (in the case of Yes in ACT64), in ACT65, it saves that one time series information after outlier removal as reference time series information in the storage 14.

[0134] Subsequently, the information processing device 10 determines in ACT21 whether it has received a notification that the information processing program has ended.

[0135] If the information processing device 10 has not received a notification that the information processing program has finished (in the case of No in ACT21), it returns to the operation of collecting time-series information in ACT11.

[0136] When the information processing device 10 receives a notification that the information processing program has finished (in the case of Yes in ACT21), it terminates the operation of extracting reference time-series information.

[0137] If the information processing device 10 detects that there are two time-series information sets after outlier removal (in the case of Yes in ACT63), in ACT66, it randomly selects one of the two time-series information sets after outlier removal as the reference time-series information and stores it in the storage 14.

[0138] Subsequently, the information processing device 10 performs the operation of determining whether it has received the termination notification of the information processing program in ACT21. The operation in ACT21 is as described above.

[0139] If the information processing device 10 detects that there are three or more time series data after outlier removal (in the case of Yes in ACT62), in ACT67, it calculates statistics for each time series data after outlier removal from the distance and similarity after outlier removal. These statistics may be, for example, the mean, median, minimum, or maximum value.

[0140] Next, in ACT20, the information processing device 10 extracts reference time series information from the time series information based on the statistical values ​​of each time series information after outlier removal and stores it in the storage 14.

[0141] Subsequently, the information processing device 10 performs the operation of determining whether it has received the termination notification of the information processing program in ACT21. The operation in ACT21 is as described above.

[0142] According to the sixth example of operation, the impact of outliers can be mitigated, and reference time series information can be extracted.

[0143] (Example 7 of operation) Next, with reference to Figure 17, a seventh example of operation of the information processing device 10 will be described. Figure 17 is a flowchart of the seventh example of operation of the information processing device 10.

[0144] The flowchart in Figure 17 is an example where the operation of ACT71 is performed between the operations of ACT20 and ACT21, compared to the flowchart in Figure 2. Below, we will explain the differences from the first operation example, namely the operation of ACT71.

[0145] In the seventh example of operation, after the operation of ACT20, the information processing device 10 displays on the display (output device 16) the results of the work time measurement performed using the reference time series information in ACT71, along with the settings used when the reference time series information was extracted. The settings used when the reference time series information was extracted include, for example, the number of time series information items, the comparison target, the calculation method for distance and similarity, the type of distance and similarity, and the type of statistical measure.

[0146] According to the seventh example of operation, the settings for extracting reference time series information can be adjusted while checking the measurement results of work time and the settings for extracting reference time series information on the display, allowing for the extraction of more effective reference time series information.

[0147] In the embodiment, an example was described in which the target of the information processing device 10 is work on a production line. However, the target of the information processing device 10 in the embodiment is not limited to this. For example, the target of the information processing device 10 in the embodiment may be the work of employees working in the back room of a store using a POS system. For example, the work may be the work of taking incoming goods out of boxes, sorting the taken-out goods, displaying the goods, etc.

[0148] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. The invention described in the original claims of this application is listed below. [1] An interface connected to a sensor that acquires video footage of the work, A processor that acquires the video via the interface and performs calculations, It has, The aforementioned processor, From the aforementioned video, time-series information including the features of the aforementioned work is collected, Clustering is applied to the aforementioned time series information to classify the time series information into a first class. Of the first class, the second class which includes three or more of the aforementioned time-series information, For each time series information within the second class, calculate at least one of the distance and similarity with the comparison target. For each time series information within the second class, a statistic is calculated from at least one of the distance and the similarity. Based on the statistics of each time series information within the second class, reference time series information to which division information indicating the division point of the work should be added is extracted from the time series information within the second class. Information processing device. [2] The aforementioned processor, Calculate at least one of the distance and the similarity between the time-series information, For each time series of information, the statistic is calculated from at least one of the distance and similarity between the time series of information, Based on the aforementioned statistics for each time series information, the reference time series information is extracted from the aforementioned time series information. [1] The information processing device described above. [3] The aforementioned processor, Reference time-series information is generated from the aforementioned time-series information, Calculate at least one of the distance and the similarity between each time series information and the reference time series information. For each time series information, the statistic is calculated from at least one of the distance and similarity between the reference time series information and each time series information. Based on the aforementioned statistics for each time series information, the reference time series information is extracted from the aforementioned time series information. [1] The information processing device described above. [4] The aforementioned processor, Each time clustering is applied to the aforementioned time-series information, the clustering results are saved. The current clustering results and the change in past clustering results are calculated, The information processing apparatus according to [1], wherein when the amount of change is greater than or equal to a threshold, the apparatus extracts the reference time series information from the time series information in the second class based on at least one of the amount of change and the statistic. [5] The aforementioned processor, Each time clustering is applied to the aforementioned time-series information, the clustering results are saved. If the second class of the current clustering result is the same as the second class of the past clustering result, the reference time series information of the second class of the past clustering result is used as the reference time series information of the second class of the current clustering result. [1] The information processing device described above. [6] An interface connected to a sensor that acquires video footage of the work, A processor that acquires the video via the interface and performs calculations, An information processing program for a computer having, The aforementioned processor, From the aforementioned video, time-series information including the features of the aforementioned work is collected. Clustering is applied to the aforementioned time series information to classify the time series information into a first class. For a second class of the first class that includes three or more of the aforementioned time series information, the distance and similarity between each time series information within the second class and the comparison target are calculated. For each time series information within the second class, a statistic is calculated from at least one of the distance and the similarity. Based on the statistics of each time series information within the second class, reference time series information to which division information indicating the division point of the work should be added is extracted from the time series information within the second class. Information processing program. [Explanation of Symbols]

[0149] 10...Information processing device, 11...Processor, 12...ROM, 13...RAM, 14...Storage, 15...Input device, 16...Output device, 17...Sensor interface, 18...Bus, 20...Sensor, S1...Comparison target, S2...Time series information.

Claims

1. An interface connected to a sensor that acquires video footage of the work, A processor that acquires the video via the interface and performs calculations, It has, The aforementioned processor, From the aforementioned video, time-series information is collected by arranging the feature vectors extracted for each frame of the video in the time direction. Clustering is applied to the aforementioned time-series information to classify the time-series information into a first class. Of the first class, the second class which includes three or more of the aforementioned time-series information, The distance and similarity between each time series information within the second class and a comparison target, which is other time series information within the second class or a reference time series information calculated from the time series information within the second class, are calculated using DTW (Dynamic Time Warping). For each time series of information within the second class, a statistic is calculated from at least one of the distance and the similarity, which is either the mean, median, minimum, or maximum value. Based on the statistics of each time series information within the second class, reference time series information to which division information indicating the division point of the work should be added is extracted from the time series information within the second class. The division information is added to the aforementioned reference time series information, The work time is measured by measuring the work time for each of the multiple tasks obtained by dividing the work at the division points, based on the time-series information associated with the reference time-series information to which the division information is attached. Information processing device.

2. The aforementioned processor, Other time-series information within the second class is used as the comparison target, The distance and the similarity between multiple time-series information within the second class are calculated, For each time series information within the second class, the statistic is calculated from at least one of the distance and similarity between the time series information. Based on the statistics of each time series information within the second class, the reference time series information is extracted from the time series information within the second class. The information processing apparatus according to claim 1.

3. The aforementioned processor, The reference time series information to be used for comparison is generated from the time series information within the second class. The distance and the similarity between each time series information in the second class and the reference time series information are calculated, For each time series information within the second class, the statistic is calculated from at least one of the distance and similarity between the reference time series information and each time series information within the second class. Based on the statistics of each time series information within the second class, the reference time series information is extracted from the time series information within the second class. The information processing apparatus according to claim 1.

4. The aforementioned processor, Each time clustering is applied to the aforementioned time-series information, the clustering results are saved. The current clustering results and the change in past clustering results are calculated, When the amount of change is greater than or equal to a threshold, the reference time series information is extracted from the time series information in the second class based on at least one of the amount of change and the statistic. The information processing apparatus according to claim 1.

5. The aforementioned processor, Each time clustering is applied to the aforementioned time-series information, the clustering results are saved. If the second class of the current clustering result is the same as the second class of the past clustering result, the reference time series information of the second class of the past clustering result is used as the reference time series information of the second class of the current clustering result. The information processing apparatus according to claim 1.

6. An interface connected to a sensor that acquires video footage of the work, A processor that acquires the video via the interface and performs calculations, An information processing program for a computer having, The aforementioned processor, From the aforementioned video, time-series information is collected by arranging the feature vectors extracted for each frame of the video in the time direction. Clustering is applied to the aforementioned time-series information to classify the time-series information into a first class. Of the first class, the second class which includes three or more of the aforementioned time-series information, The distance and similarity between each time series information within the second class and a comparison target, which is either another time series information within the second class or a reference time series information calculated from the aforementioned time series information within the second class, are calculated using DTW (Dynamic Time Warping). For each time series of information within the second class, a statistic that is either the mean, median, minimum, or maximum is calculated from at least one of the distance and the similarity. Based on the statistics of each time series information within the second class, reference time series information to which division information indicating the division point of the work should be added should be extracted from the time series information within the second class. The division information is added to the aforementioned reference time series information. The work time is measured by dividing the work at the division points and measuring the work time for each of the multiple tasks obtained based on the time-series information associated with the reference time-series information to which the division information is attached. Information processing program.

Images