Information processing device and program

Abstract

To improve the accuracy of work time measurement.SOLUTION: According to an embodiment, an information processing device includes an extraction unit, an acquisition unit, an association unit, and a measurement unit. The extraction unit extracts a working image during working in which a series of a plurality of types of work is performed from an image obtained by consecutively imaging movements of a worker. The acquisition unit acquires time series information representing characteristics of movement of the worker at each time point in a time series order from each working image. The association unit compares sample time series information in which a work type of work and a switching timing of the work are known with the time series information, and associates time points at which movement characteristics are similar. The measurement unit measures a work time of each work type about the time series information on the basis of an association result.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] Conventionally, there are known devices that measure the working time of workers on a production line. For example, such devices use a camera to identify the area where the worker is present, and when the worker is present in an area where a work object is present, the time the worker spends there is measured as the working time.

[0003] However, in the conventional task time measurement, when the relative positional relationship between the camera and the worker changes, it may be determined that the area where the work object exists does not correspond to the area where the worker exists, even though the worker is working. In this case, the task time is not measured correctly. In other words, there is room for improvement in the accuracy of task time measurement. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to improve the accuracy of task time measurement. [Means for solving the problem]

[0005] An information processing device according to an embodiment includes an extraction unit, an acquisition unit, a matching unit, and a measurement unit. The extraction unit extracts images of a series of tasks of a plurality of task types from images captured continuously of the worker's movements. The acquisition unit acquires time-series information representing the characteristics of the worker's movements at each point in time in chronological order from each of the images of the worker movements. The matching unit compares the time-series information with model time-series information in which the task types and the switching times of the tasks are known, and matches the time-series information with the time-series information at which the characteristics of the movements are similar. The measurement unit measures the work time for each task type for the time-series information based on the matching result. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an activity time measurement system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of a CPU of the task time measurement device according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a process for acquiring a work video according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a process for acquiring time-series data according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a storage process of time-series data according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a selection process of target data and example data according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the model data extraction process according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a process for associating target data with example data according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a process for estimating a task type and a timing at which the task type is switched according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the task time measurement process according to the embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a process executed by the task time measurement device according to the embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a process executed by the activity time measurement device according to the first modification. [Figure 13] FIG. 13 is a diagram illustrating an example of a process executed by the activity time measuring device according to the second modification. [Figure 14] FIG. 14 is a diagram illustrating an example of a process executed by an activity time measuring device according to the third modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an embodiment of an information processing device and a program will be described in detail with reference to the accompanying drawings. In the following embodiment, an example will be described in which the information processing device and the program of the present invention are applied to measuring the working time of workers who perform various tasks on a production line, but the present invention is not limited to this embodiment.

[0008] 1 is a block diagram showing an example of the configuration of an information processing system S according to this embodiment. The work management system 1 has an activity time measurement device 10, a work information server 20, and a network camera 30. The activity time measurement device 10, the work information server 20, and the network camera 30 are connected via a network such as the Internet or a LAN (Local Area Network).

[0009] The work information server 20 is a server device that manages information related to work by workers. For example, it manages information related to the work time of each worker. In addition, for example, the work information server 20 stores information related to standard work time for each type of work.

[0010] The network camera 30 is a camera that can be connected to a network such as the Internet or a LAN. There may be provided a plurality of network cameras 30. The network cameras 30 are provided at positions where they can capture images of workers working on the production line.

[0011] For example, the network camera 30 captures images of workers working on a production line. By using the network camera 30, it is possible to capture images of workers working in locations far from the installation position of the work time measurement device 10 (described later). Images (video) captured by the network camera 30 include the time when the image was captured.

[0012] The task time measurement device 10 is a device that measures the task time of a worker. For example, the task time measurement device 10 measures the task time of a worker on a production line. The configuration of the task time measurement device 10 will be described below with reference to Figs. 2 and 3.

[0013] First, a description will be given of the hardware configuration of the activity time measurement device 10. FIG 2 is a block diagram showing an example of the hardware configuration of the activity time measurement device 10.

[0014] 2, the operation time measurement device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a communication I / F (Interface) 14, a camera I / F 15, and a storage unit 16. These units 11 to 16 are connected to each other via a data bus.

[0015] Furthermore, the operation time measurement device 10 is connected to a camera 17 via a camera I / F 15. The camera 17 has a built-in RTC (Real Time Clock) 18. The RTC 18 keeps track of the current date and time.

[0016] The CPU 11 comprehensively controls the task time measurement device 10. For example, the CPU 11 uses the RAM 12 as a working area and executes various processes by executing programs stored in the ROM 13, the storage unit 16, etc. The storage unit 16 is a non-volatile memory to which data can be written and rewritten. For example, the storage unit 16 stores target data 161 and model data 162, which will be described later.

[0017] The communication I / F a4 is a device for establishing a predetermined type of communication with the work information server 20 and the network camera 30. The communication type is not particularly limited, but examples include those conforming to standards such as Ethernet (registered trademark) and IEEE802.11.

[0018] Camera I / F 15 controls camera 17 according to instructions from CPU 11, captures an image signal output from camera 17, and outputs it as a captured image. Camera 17 captures an image according to the control of camera I / F 15, and outputs an image signal. The number of cameras 17 connected to operation time measurement device 10 may be one or more. Camera 17 is provided at a position where it can capture an image of a worker performing work on the production line.

[0019] For example, the camera 17 captures images of workers working on a production line. The images (moving images) captured by the camera 17 are recorded with the time when the images were captured using the RTC 18.

[0020] The task time measurement device 10 also includes functional units realized by cooperation between the CPU 11 and programs stored in the ROM 13 and the storage unit 16. Specifically, as shown in Fig. 2, the CPU 11 executes the programs to realize functional units such as a first acquisition unit 111, a detection unit 112, a generation unit 113, a second acquisition unit 114, a storage control unit 115, a first selection unit 116, a second selection unit 117, an association unit 118, an estimation unit 119, and a measurement unit 120 on the RAM 12. The storage unit 16 also stores target data 161 and model data 162.

[0021] The first acquisition unit 111 acquires a plurality of images capturing work performed by a worker. For example, the first acquisition unit 111 acquires a video captured by the camera 17. Also, for example, the first acquisition unit 111 acquires a video captured by the network camera 30. Note that the first acquisition unit 111 may acquire only one of the video captured by the camera 17 and the network camera 30.

[0022] The detection unit 112 detects a work start trigger or a work end trigger. For example, the detection unit 112 detects a work start trigger or a work end trigger from a moving image acquired by the first acquisition unit 111. The work start trigger is, for example, a specific posture that a worker takes when starting work. The work end trigger is, for example, a specific posture that a worker takes when ending work.

[0023] In this embodiment, the work start trigger is described as the worker taking a specific posture, but the work start trigger is not limited to this. For example, the work start trigger may be provided in the work time measurement device 10 and may be detected as the worker pressing the work start input button.

[0024] The work start trigger may be, for example, detecting a sound notifying the start of work by recording audio simultaneously with video images by the camera 17 or the network camera 30. The work start trigger may be, for example, detecting the passage of a predetermined work start time.

[0025] In this embodiment, the work end trigger is described as the worker taking a specific posture different from the work start trigger, but the work end trigger is not limited to this. For example, the work end trigger may be provided in the work time measurement device 10 and the worker pressing the work end input button may be detected.

[0026] The work end trigger may be, for example, detecting a sound notifying the end of work by recording audio simultaneously with video images by the camera 17 or the network camera 30. The work end trigger may be, for example, detecting the passage of a predetermined work end time.

[0027] Furthermore, for example, when the detection unit 112 detects a work start trigger from the video acquired by the first acquisition unit, the detection unit 112 records a work start time in the video. Similarly, when the detection unit 112 detects a work end trigger from the video acquired by the first acquisition unit, the detection unit 112 records a work end time in the video.

[0028] The generating unit 113 generates one cycle of work moving images (hereinafter, one cycle moving images) by extracting moving images during work in which a series of work of a plurality of work types is performed from moving images of the worker's actions captured by the camera 17 or the network camera 30. The generating unit 113 is an example of an extracting unit. The one cycle moving images are also an example of images during work.

[0029] For example, when both the work start time and the work end time are recorded in the moving image by the detection unit 112, the generation unit 113 generates a one-cycle moving image based on the work start time and the work end time. An example of the generation process of the one-cycle moving image will be described below with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of the generation process of the one-cycle moving image.

[0030] 3, at time t+0, the detection unit 112 detects a work start trigger and records the work start time. Similarly, at time t+3, the detection unit 112 detects a work end trigger and records the work end time. The generation unit 113 generates a moving image from the work start time (t+1) to the work start time (t+2) as one-cycle moving image Pa.

[0031] In the one-cycle moving image Pa of FIG. 3, it is recorded that the worker is performing task A at time t+2, and is performing task B at time t+3. Note that task A and task B represent task types. In FIG. 3, the worker is performing two task types, task A and task B, but the task types are not limited to two. For example, the task types may be three or more.

[0032] 3, at time t+5, the detection unit 112 detects a work start trigger and records the work start time. Similarly, at time t+9, the detection unit 112 detects a work end trigger and records the work end time. The generation unit 113 generates a video from the work start time (t+6) to the work start time (t+8) as one-cycle video Pb.

[0033] In the one-cycle moving image Pb in FIG. 3, it is recorded that the worker is performing task A at time t+6, and is performing task B at times t+7 and t+8.

[0034] The video acquired by the first acquisition unit 111 also records scenes (times t+4 and t+10) of the worker performing activities that are not included in the work and are to be recorded for the performed work, but these are not included in the one-cycle video images Pa and Pb. In this way, the generation unit 113 can generate a video that records only scenes of the worker performing activities that are included in the work.

[0035] 2, the description will continue. The second acquisition unit 114 acquires time-series data that represents the characteristics of the worker's actions at each time point in chronological order. The time-series data is an example of time-series information.

[0036] For example, the second acquisition unit 114 extracts a feature vector for each frame by applying feature extraction to each image constituting one cycle of a moving image. The second acquisition unit 114 also acquires time-series data by arranging the extracted feature vectors for each frame in the time direction. For feature extraction, known techniques such as human behavior recognition, skeletal detection, object detection, and vectorization of the image itself can be appropriately used.

[0037] The second acquisition unit 114 may store face images of the workers in advance and identify the workers appearing in the video by using a known image recognition technique. In this case, the second acquisition unit 114 may associate information for identifying the identified workers with the time-series data.

[0038] The process of acquiring time series data will be described below with reference to Fig. 4. Fig. 4 is a diagram for explaining an example of the process of acquiring time series data.

[0039] 4 includes frame Fa, which is an image captured at time t+1, through frame Fk, which is an image captured at time t+9. The second acquisition unit 114 applies feature extraction to each of the frames Fa through Fk, and extracts feature vectors Ba through Bk from each of the frames Fa through Fk. Next, the second acquisition unit 114 acquires time-series data Dx by arranging the feature vectors Ba through Bk in chronological order.

[0040] In Fig. 4, it is assumed that the worker is performing task A in frames Fa to Fc (times t+0 to t+2) and Fg (time t+6). It is assumed that the worker is away from the work space (away from his / her desk) in frames Fd to Ff (times t+3 to t+5). It is assumed that the worker is performing task B in frames Fi to Fk (times t+7 to t+10).

[0041] It is considered that at least one of the following is different in the captured image: the worker's position, the worker's posture, the position of the object being worked on, the angle of the object, etc., between Task A, Leaving the Desk, and Task B. From this, it is considered that the characteristic vectors Ba to Bc, Bg, the characteristic vectors Be, Bf, and the characteristic vectors Bh to Bk are different in nature. Therefore, it can be said that the time series data Dx in Fig. 4 represents, in time series, the change in the image characteristics accompanying the change in the worker's activity from time t+0 to t+11.

[0042] Returning to Fig. 2, the description will be continued. The storage control unit 115 performs control to store the time series data acquired by the second acquisition unit 114 in either the target data 161 or the model data 162 of the storage unit 16. The target data 161 is an example of the first group. The model data 162 is an example of the second group.

[0043] Here, the target data 161 is a set of stored target time series data. The target time series data is, for example, time series data that is used to estimate the task type and task change time. The target time series data is an example of target time series information.

[0044] Moreover, the model data 162 is a set of stored model time series data. For example, the model time series data is time series data in which the task type and task switching time are known. Specifically, the model time series data is time series data in which the breakdown of task types is determined, such as performing task A from time t+a to t+c, switching from task A to task B at t+d, and performing task B from time t+d to t+e. The model time series data is an example of model time series information.

[0045] The storage process of time series data will be described below with reference to Fig. 5. Fig. 5 is a diagram for explaining an example of the storage process of time series data. Time series data Dx in Fig. 5 represents specific time series data acquired by the second acquisition unit 114. As an example, when the time series data Dx is time series data selected as model time series data by a user who is an administrator of the work time measurement device 10, the storage control unit 115 controls storage in the model data 162 of the storage unit 16.

[0046] Furthermore, the storage control unit 115 performs control to store the time series data Dx in the target data 161 of the storage unit 16 when the time series data Dx is time series data that has not been selected as model time series data by the user.

[0047] In this embodiment, the storage control unit 115 controls the storage of time series data in either the target data 161 or the model data 162 in accordance with the user's selection, but the method of storing the time series data in either the target data 161 or the model data 162 is not limited to the above.

[0048] For example, the storage control unit 115 may perform control to automatically store time series data recording work performed from the beginning of a day to a predetermined number of times for each of a plurality of workers in the model data 162. In addition, for example, the storage control unit 115 may perform control to automatically store time series data recording work performed by a specific worker in the model data 162.

[0049] In this embodiment, the user inputs a breakdown of task types (hereinafter also referred to as teaching data) for the model time series data stored in the model data 162. As an example, the user inputs, for the model time series data stored in the model data 162, that task A is performed from time t+a to t+c, task A is switched to task B at t+d, and task B is performed from time t+d to t+e. The storage control unit 115 records the input contents in the model time series data.

[0050] The method of acquiring the teaching data is not limited to the above. For example, the teaching data may be acquired by preparing standard work data that records standard work for each work type, and identifying the work type performed by the worker in the video based on the similarity between the feature vector of the standard work data and the feature vector of the model time-series data.

[0051] Returning to Fig. 2, the description will continue. The first selection unit 116 selects one piece of time series data from the target time series data stored in the target data 161. For example, the first selection unit 116 selects, from the target time series data stored in the target data 161, the target time series data having the earliest recorded work start time.

[0052] When the target time series data is selected, the second selection unit 117 selects one model time series data from the multiple model time series data stored in the model data 162. For example, the second selection unit 117 first selects multiple model time series data from the multiple model time series data stored in the model data 162.

[0053] As an example, the second selection unit 117 refers to the work start times recorded in the multiple model time series data, and selects a predetermined number of model time series data in order of proximity to the work start time recorded in the target time series data selected by the first selection unit 116. Note that the criteria for the multiple model time series data are not limited to the above. For example, the second selection unit 117 may randomly select a predetermined number of model time series data.

[0054] Next, the second selection unit 117 selects, from among the multiple selected model time series data, the model time series data that has the smallest association cost when associated with the target time series data selected by the first selection unit 116. Note that the method for associating the target time series data with the model time series data may be any known method such as DTW (Dynamic Time Warping).

[0055] The selection process of the time series data will be described below with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are diagrams for explaining an example of the selection process of the time series data. As shown in the upper part of Fig. 6, the first selection unit 116 selects the target time series data ODa from the target time series data stored in the target data 161.

[0056] 6, the second selection unit 117 selects model time series data EDa and model time series data EDb from the model time series data stored in the model data 162. In this example, the second selection unit 117 selects two model time series data, but the second selection unit 117 may select three or more model time series data.

[0057] As shown in the lower part of Figure 6, the model time series data EDa and the model time series data EDb are composed of a record of the period when the worker is performing task A (denoted as A in Figure 6), a record of the period when the worker is performing task B (denoted as B in Figure 6), and a record of the period when the worker is performing task C (denoted as C in Figure 6).

[0058] In addition, the model time series data EDa and the model time series data EDb record, as teaching data, the task switching time CPa at which the tasks performed by the worker switch from task A to task B, and the task switching time CPb at which the tasks switched from task B to task C.

[0059] The second selection unit 117 may exclude model time-series data for which no teaching data is recorded from the selection targets, even if the model time-series data is stored in the model data 162. When the second selection unit 117 selects model time-series data for which no teaching data is recorded, the second selection unit 117 may request the user to input teaching data.

[0060] Next, the second selection unit 117 calculates the distance between the target time series data ODa and each of the model time series data EDa and the model time series data EDb, as shown in Fig. 7. In Fig. 7, the distance between the target time series data ODa and the model time series data EDa is 58, and the distance between the target time series data ODa and the model time series data EDb is 100.

[0061] Here, the shorter the distance, the lower the matching cost by DTW. Therefore, in this embodiment, the second selection unit 117 extracts model time series data with the smallest distance from among the multiple selected model time series data. In the example of FIG. 7, the model time series data EDa has a lower matching cost with the target time series data ODa than the model time series data EDb. Therefore, the second selection unit 117 extracts (selects) the model time series data EDa.

[0062] Returning to Fig. 2, the explanation will be continued. The association unit 118 compares the model time-series information with the target time-series information, and associates the time points at which the characteristics of the movements of both are similar. For example, the association unit 118 associates one target time-series data selected by the first selection unit 116 with one model time-series data selected by the second selection unit 117 at the frame level using a known method such as DTW.

[0063] The association process will be described below with reference to Fig. 8. Fig. 8 is a diagram for explaining an example of the association process. It is assumed that the first selection unit 116 selects the target time series data ODa, and the second selection unit 117 extracts (selects) the target time series data ODa.

[0064] In the example of Fig. 8, one frame is represented by one circle. That is, the target time series data ODa is composed of 13 frames. On the other hand, the model time series data EDa is composed of 11 frames. The association unit 118 associates the target time series data ODa and the model time series data EDa at the frame level using DTW. As shown in Fig. 8, the association unit 118 can associate time series data with different numbers of frames with each other.

[0065] Returning to Fig. 2, the explanation will be continued. The estimation unit 119 estimates the task period and task switching time for each task type of the target time series data based on the association result by the association unit 118. The estimation process will be explained below with reference to Fig. 9. Fig. 9 is a diagram for explaining an example of the estimation process. As a premise, the first circle in the target time series data ODa and the model time series data EDa represents the frame corresponding to the task start time, and the last circle represents the frame corresponding to the task end time.

[0066] First, the estimation unit 119 estimates the frame indicated by (1) corresponding to the task change time CPa as the estimated change time EPa. Similarly, the estimation unit 119 estimates the frame indicated by (2) corresponding to the task change time CPb as the estimated change time EPb. Note that, if there are multiple frames corresponding to the task change time, the midpoint between the first frame and the last frame among the corresponding frames may be estimated as the estimated change time.

[0067] 9, the estimation unit 119 estimates the period from the task start time to the estimated switching time EPa as an estimated period EA that is estimated to be the execution period of task A. The estimation unit 119 also estimates the period from the estimated switching time EPa to the estimated switching time EPb as an estimated period EB that is estimated to be the execution period of task B. The estimation unit 119 also estimates the period from the estimated switching time EPb to the task end time as an estimated period EC that is estimated to be the execution period of task C.

[0068] Returning to Fig. 2, the explanation will be continued. The measurement unit 120 measures the work time for each work type based on the association result by the association unit 118. The measurement unit 120 also transmits the measurement result to the work information server 20 via the communication I / F. The measurement process will be explained below with reference to Fig. 10. Fig. 10 is a diagram for explaining an example of the measurement process.

[0069] 10, the measurement unit 120 measures the lengths of the estimated periods EA, EB, and EC estimated by the estimation unit 119 as the work time of task A, the work time of task B, and the work time of task C, respectively. In the example of Fig. 10, the measurement unit 120 measures that the work time of task A is 15.5 seconds, the work time of task B is 54.2 seconds, and the work time of task C is 8.9 seconds.

[0070] Next, a description will be given of the processing executed by the activity time measurement device 10. Fig. 11 is a flowchart showing an example of the processing executed by the activity time measurement device 10.

[0071] First, the first acquisition unit 111 acquires a camera image recording an operation performed by a worker (step S101). For example, the first acquisition unit 111 acquires a video from at least one of the camera 17 and the network camera 30. The video includes a time when the video was captured.

[0072] Next, the detection unit 112 determines whether or not a work start trigger is detected from the moving image acquired in step S101 (step S102). If a work start trigger is not detected (step S102: No), the process proceeds to step S104.

[0073] When a work start trigger is detected (step S102: Yes), the detection unit 112 records the time when the work start trigger is detected in the video acquired in step S101 as the work start time in the video (step S103).

[0074] Next, the detection unit 112 determines whether or not a work end trigger is detected from the moving image acquired in step S101 (step S104). If a work end trigger is not detected (step S104: No), the process returns to step S101.

[0075] When a work end trigger is detected (step S104: Yes), the detection unit 112 records the time when the work end trigger is detected in the video acquired in step S101 as the work end time in the video (step S105).

[0076] Next, the generating unit 113 determines whether both the work start time and the work end time are recorded in the video acquired in step S101 (step S106). If at least one of the work start time and the work end time is not recorded (step S106: No), the process returns to step S101. If both the work start time and the work end time are recorded (step S106: Yes), the generating unit 113 generates one cycle of the video (step S107).

[0077] Next, the second acquisition unit 114 acquires time-series data based on the one-cycle video image (step S108). For example, the second acquisition unit 114 acquires time-series data by extracting a feature vector for each frame from the one-cycle video image and arranging the extracted feature vectors for each frame in the time direction.

[0078] Next, the storage control unit 115 adds the time series data acquired in step S108 to the target data 161 or the model data 162 (step S109). For example, the storage control unit 115 stores the time series data selected by the user as the model time series data in the model data 162. In addition, the storage control unit 115 stores the time series data not selected as the model time series data in the target data 161.

[0079] Next, the storage control unit 115 determines whether or not time-series data has been added to the model data 162 (step S110). If time-series data has not been added to the model data 162, in other words, if time-series data has been added to the target data 161 (step S110: No), the process proceeds to step S112.

[0080] On the other hand, when the time-series data is added to the model data 162 (step S110: Yes), the storage control unit 115 acquires the training data (step S111). For example, the storage control unit 115 accepts the input of the training data from the user and acquires the input content as the training data.

[0081] Next, the first selection unit 116 determines whether one or more time-series data exists in the target data 161 and two or more time-series data exists in the example data (step S112). If no time-series data exists (step S112: No), the process returns to step S101.

[0082] On the other hand, if time series data exists (step S112: Yes), the first selection unit 116 selects one piece of target time series data from the target data 161 (step S113). For example, the first selection unit 116 selects the target time series data that records the earliest operation among the target time series data stored in the target data 161.

[0083] Next, the second selection unit 117 selects a plurality of target time series data from the model data 162 (step S114). For example, the second selection unit 117 refers to the times recorded in the plurality of model time series data stored in the model data 162, and selects a predetermined number of model time series data in order of the recorded times closest to the times recorded in the target time series data selected in step S113.

[0084] Next, the second selection unit 117 calculates the distance between the one target time series data selected in step S113 and each of the model time series data selected in step S114 (step S115).The second selection unit 117 extracts model time series data from the selected model time series data with the minimum distance calculated in step S115 (step S116).

[0085] Next, the association unit 118 associates the target time series data selected in step S113 with the model time series data selected in step S116 (step S117). For example, the association unit 118 associates the target time series data with the model time series data at the frame level using DTW.

[0086] Next, the estimation unit 119 estimates the task switch time and task type of the target time series data (step S118). For example, the estimation unit 119 estimates the task switch time and task type for each frame based on the association result of step S117. Next, the measurement unit 120 measures the task time (step S119). For example, the measurement unit 120 measures the task time for each task type based on the estimation result of step S118.

[0087] Next, the first acquisition unit 111 determines whether an end instruction to end the process has been received from the user (step S120). If an end instruction has not been received (step S120: No), the process returns to step S101. If an end instruction has been received (step S120: Yes), the process ends.

[0088] As described above, the work time measurement device 10 according to this embodiment acquires time series data based on multiple images recording multiple tasks related to multiple work types performed by a worker, selects one target time series data from multiple target time series data that are the subject of work time measurement, and when the target time series data is selected, selects one model time series data from multiple model time series data for which the work times are known, matches the two selected data, and measures the work time for each work type for the target time series data based on the matching result.

[0089] In this way, the work time measurement device 10 according to this embodiment associates the target time series data with the model time series data, and measures the work time for each task type from the association result. Therefore, even if the angle of view of the camera that captured the video on which the time series data is based has changed, it is possible to estimate the work time for each task type of the target time series data by associating the target time series data with the model time series data. Therefore, even in such a case, the work time measurement device 10 according to this embodiment can measure the work time for each task type with high accuracy. In other words, the work time measurement device 10 according to this embodiment can improve the accuracy of work time measurement.

[0090] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each device of the information processing system S. Therefore, some modified examples of the above-described embodiment will be described below as other embodiments. The following mainly describes the differences from the above-described embodiment, and a detailed description of the commonalities with the contents already described will be omitted. The modified examples described below may be implemented individually or in appropriate combination.

[0091] (Variation 1) In the above embodiment, a configuration has been described in which a feature vector for each frame is extracted using techniques such as human behavior recognition, skeleton detection, object detection, vectorization of the image itself, etc. In this modified example, a configuration will be described in which a relative distance is calculated for each position of features that can be expressed by coordinates on an image, such as a skeleton position of a worker or an object position, and the calculated relative distance is added to the feature vector as a new feature element.

[0092] In this modification, the second acquisition unit 114 calculates the relative distance between multiple feature points, such as the positions of the worker's skeleton and the positions of objects, on each image (frame) constituting one cycle of the moving image generated by the generation unit 113. The second acquisition unit 114 adds a vector representing the calculated relative distance as a new feature element to the feature vector for each frame extracted by applying feature extraction. The second acquisition unit 114 acquires time series data by arranging the feature vectors to which the new feature element has been added in chronological order.

[0093] The process executed by the work time measurement device 10 according to this modification will be described below. Fig. 12 is a flowchart showing an example of the process executed by the work time measurement device 10 according to modification 1. Steps S201 to S207 are similar to steps S101 to S107 in Fig. 11, and therefore description thereof will be omitted.

[0094] After step S207, the second acquisition unit 114 applies feature extraction to each image constituting the one-cycle video sequence generated in step S207 to extract a feature vector for each frame (step S208). Next, the second acquisition unit 114 calculates the relative distance between multiple feature points, such as the position of the worker's skeleton and the position of an object on each image constituting the one-cycle video sequence, which are feature elements of the extracted feature vector (step S209).

[0095] Next, the second acquisition unit 114 acquires time series data (step S210). For example, the second acquisition unit 114 adds the vector representing the relative distance calculated in step S209 as a new feature element to the extracted feature vector, and arranges the feature vector to which the new feature element has been added in chronological order to acquire time series data. Steps S211 to S222 are similar to steps S109 to S120 in FIG. 11, and therefore description thereof will be omitted.

[0096] In this modification, the time series data includes information representing the relative distance between feature points as an element. Therefore, the task time measurement device 10 according to this modification can obtain position information of a tool, such as a screwdriver, that is installed in a work space and has a fixed use location, and can determine that the screwdriver has been used when the relative distance between the worker's hand and the screwdriver becomes small, even if the worker's position on the image changes slightly. This allows the task time measurement device 10 according to this modification to estimate the period during which the worker is performing a task, and also to estimate the task switching time and each task type.

[0097] (Variation 2) In the above embodiment, the second selection unit 117 selects a plurality of model time series data from the model data 162 based on the time recorded in the target time series data and the time recorded in the model time series data. In this modification, a description will be given of a form in which a plurality of model time series data is selected from the model data 162 by using information related to the work.

[0098] In this modification, the one-cycle video generated by the generating unit 113 is stored in the work information server 20. Furthermore, the one-cycle video is associated with work information that is information related to the work.

[0099] The work information is, for example, a product ID that identifies the product related to the performed work, a worker ID that identifies the worker who performed the work, the employment history of the worker who performed the work, the product ID of the product that the worker has work experience with, the work speed for each work type, etc. As an example, the work information server 20 accepts input of work information from an administrator of the work information server 20 or the like, and stores the work information in a storage device provided in the work information server 20 in association with one cycle of the moving image.

[0100] In addition, the work information server 20 may store information such as a facial image, worker ID, employment history, product ID of products for which the worker has work experience, and work speed for each type of work in advance for each worker, and acquire work information based on this information.

[0101] For example, the work information server may extract a facial image of a worker appearing in the one-cycle video using a known image recognition technology or the like from the one-cycle video, and acquire work information based on information corresponding to the facial image. In this case, by storing information in advance that associates a product image with a product ID, the product related to the work can also be identified from the one-cycle video.

[0102] In this modification, the second acquisition unit 114 acquires work information from the work information server 20 and performs a process of linking the work information to the acquired time series data. The second selection unit 117 compares the work information of the target time series data selected by the first selection unit 116 with the work information of the multiple model time series data stored in the model data 162. The second selection unit 117 then selects a predetermined number of model time series data in descending order of the similarity of the work information (the degree of agreement of information included in the work information).

[0103] The process executed by the work time measurement device 10 according to this modified example will be described below. Fig. 13 is a flowchart showing an example of the process executed by the work time measurement device 10 according to modified example 2. Steps S301 to S307 are similar to steps S101 to S107 in Fig. 11, and therefore their description will be omitted. However, in this modified example, the one-cycle moving image generated in step S307 is stored in the work information server 20.

[0104] After step S307, the second acquisition unit 114 acquires work information to be linked to the time-series data (step S308). For example, the second acquisition unit 114 acquires work information associated with the one-cycle moving image stored in the work information server 20 as the work information to be linked to the time-series data.

[0105] 11, but in this modification, the second acquisition unit 114 acquires time series data from one cycle of moving images corresponding to the work information acquired in step S308. Then, the second acquisition unit 114 links the time series data to the work information acquired in step S308. Steps S310 to S314 are similar to steps S109 to S113 in FIG. 11, and therefore description thereof will be omitted.

[0106] After step S314, the second selection unit 117 calculates the similarity between each of the multiple model time-series data stored in the model data 162 and the target time-series data selected in step S314 (step S315).

[0107] Next, the second selection unit 117 selects a plurality of model time series data based on the similarity calculated in step S315 (step S316). For example, the second selection unit 117 selects a predetermined number of model time series data in descending order of similarity with the target time series data. Steps S317 to S322 are similar to steps S115 to S120 in FIG. 11, and therefore will not be described.

[0108] According to this modification, when selecting a plurality of model time series data from the model time series data stored in the model data 162, model time series data having similar task information to the selected target time series data can be selected. For example, if the worker related to the target time series data and the worker related to the model time series data are the same, there is a high possibility that the characteristics of the two are similar. Therefore, the task time measurement device 10 according to this modification can increase the possibility of selecting model time series data having similar task information to the target time series data, thereby increasing the possibility of selecting model time series data having similar features to the target time series data. Furthermore, by selecting model time series data having similar features to the target time series data, it is expected that the accuracy of estimation of the task switching time and the task type can be improved, and therefore the task time measurement device 10 according to this modification can improve the accuracy of task time measurement.

[0109] (Variation 3) In the above embodiment, a form in which time series data is added to the model data 162 according to a selection by an administrator or the like has been described. In this modified example, a form in which target time series data added to the target data 161 is added to the model data 162 when the target time series data satisfies a predetermined condition will be described.

[0110] In this modified example, when an administrator or the like has set up automatic addition of model time series data and certain target time series data satisfies certain conditions, the memory control unit 115 controls the movement of the target time series data from the target data 161 to the model data 162.

[0111] For example, when the automatic addition setting for model time series data is ON and the difference in time length between the target time series data and the standard time stored in the work information server 20 is less than a threshold value, the memory control unit 115 moves the target time series data from the target data 161 to the model data 162.

[0112] In this embodiment, the standard time is information that indicates a standard work time for each work type. The standard time is determined for each work process, such as the assembly of product A and the assembly of product B.

[0113] Also, for example, when the automatic addition setting of model time series data is ON and the distance from all model time series data stored in model data 162 is less than a threshold, the storage control unit 115 moves the target time series data from the target data 161 to the model data 162. Note that the storage control unit 115 may move the target time series data from the target data 161 to the model data 162 when at least one of the difference in time length from the standard time and the distance from the model time series data is less than a threshold.

[0114] The following describes the processing executed by the work time measurement device 10 according to this modified example. Fig. 14 is a flowchart showing an example of the processing executed by the work time measurement device 10 according to modified example 3. Steps S401 to S419 are similar to steps S101 to S119 in Fig. 11, and therefore description thereof will be omitted.

[0115] After step S419, the storage control unit 115 determines whether the automatic addition setting of the model time series data is ON (step S420). If the automatic addition setting is not ON (step S420: No), the process proceeds to step S424. If the automatic addition setting is ON (step S420: Yes), the storage control unit 115 acquires the standard time of the work process related to the target time series data from the work server (step S421).

[0116] Next, the memory control unit 115 determines whether at least one of the difference in time length from the standard time and the distance from the example time series data is less than a threshold (step S422). If both are equal to or greater than the threshold (step S422: No), the process proceeds to step S424.

[0117] On the other hand, if at least one of them is less than the threshold (step S422: Yes), the target time-series data for which the task time was measured in step S419 is moved from the target data 161 to the example data 162 (step S423). Step S424 is similar to step S120 in FIG. 11, and therefore a description thereof will be omitted.

[0118] According to this modification, it is possible to treat more time series data as model time series data. This increases the possibility of selecting model time series data that is highly similar to the target time series data when selecting model time series data. In other words, the task time measurement device 10 according to this modification can improve the accuracy of task time measurement.

[0119] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0120] 10. Working time measurement device 11 CPU 12 RAM 13 ROM 14 Communication I / F 15 Camera I / F 16 Memory section 17 Camera 18 RTC 20 Work Information Server 30 Network Camera 111 First acquisition part 112 Detection unit 113 Generation part 114 Second Acquisition Department 115 Memory control unit 116 First Selection Section 117 Second Selection Section 118 Mapping section 119 Estimation Department 120 Measurement section [Prior art documents] [Patent documents]

[0121] [Patent Document 1] JP 2020-135767 A

Claims

1. an extracting unit that extracts images of a series of operations performed by a worker from images captured continuously of the worker's movements; an acquisition unit that acquires time-series information representing the characteristics of the worker's actions at each time point in chronological order from each of the in-work images; a matching unit that compares the time-series information with example time-series information in which the task type and the task switching time are known, and matches the time points at which the motion characteristics are similar; A measurement unit that measures an operation time for each operation type for the time-series information based on a result of the association by the association unit; An information processing device comprising:

2. a first selection unit that selects one of the target time series information from among the acquired time series information, the target time series information being used to estimate the task types of the multiple tasks and the switching times of the tasks; A second selection unit that selects one example of time-series information from the plurality of example time-series information, the associating unit associates a time point at which the motion characteristics of the selected target time-series information and the selected model time-series information are similar; The measurement unit measures the task time for each task type for the target time-series information based on the association result. The information processing device according to claim 1 .

3. The acquisition unit extracts a feature vector from the in-work image, and acquires the time-series information based on the extracted feature vector.

3. The information processing device according to claim 1 or 2.

4. the acquisition unit extracts a plurality of feature positions on the work image, calculates a relative distance between the feature positions, and adds a vector representing the relative distance to the feature vector as a new feature element; The information processing device according to claim 3 .

5. the acquiring unit associates the target time-series information and the model time-series information with work information including at least one of identification information of a worker related to the target time-series information or the model time-series information, the work experience of the worker, the work time of the worker, and identification information of a product related to the time-series information; The second selection unit selects the example time-series information based on a similarity between the target time-series information selected by the first selection unit and the task information linked to the target time-series information. The information processing device according to claim 2 .

6. a storage control unit that determines whether the time series information is the target time series information or the model time series information, and stores the time series information determined to be the target time series information as a first group and the time series information determined to be the model time series information as a second group in a storage device; The first selection unit selects one of the target time-series information from the first group; the second selection unit, when a plurality of pieces of model time-series information are stored as the second group, compares the target time-series information selected by the first selection unit with the plurality of pieces of model time-series information, and selects one piece of model time-series information with a minimum association cost. The information processing device according to claim 2 .

7. the storage control unit, when at least one of the following is satisfied: a matching cost of the target time series information with the model time series information stored in the storage device as the second group is less than a threshold; and a difference in time length between the measured work time for each task type and a standard time for each task type is less than a threshold, performs control to move the target time series information from the first group to the second group as the model time series information. The information processing device according to claim 6.

8. Computer, an extracting unit that extracts images of a series of operations performed by a worker from images captured continuously of the worker's movements; an acquisition unit that acquires time-series information representing the characteristics of the worker's actions at each time point in chronological order from each of the in-work images; a matching unit that compares the time-series information with example time-series information in which the task type and the task switching time are known, and matches the time points at which the motion characteristics are similar; A measurement unit that measures an operation time for each operation type for the time-series information based on a result of the association by the association unit; A program that functions as a

Images