Information processing device and program

Abstract

To improve the accuracy of work time measurement.SOLUTION: According to an embodiment, an information processing device includes an extraction unit, an acquisition unit, an association unit, and a measurement unit. The extraction unit extracts a working image during working in which a series of a plurality of types of work is performed from an image obtained by consecutively imaging movements of a worker. The acquisition unit acquires time series information representing characteristics of movement of the worker at each time point in a time series order from each working image. The association unit compares sample time series information in which a work type of work and a switching timing of the work are known with the time series information, and associates time points at which movement characteristics are similar. The measurement unit measures a work time of each work type about the time series information on the basis of an association result.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus and a program.

Background Art

[0002] Conventionally, an apparatus for measuring the working hours of workers on a production line has been known. As such an apparatus, for example, there is one that uses a camera to identify the area where a worker is present and measures the stay time as the working hours when the worker is present in the area where the work object is present.

[0003] However, in conventional working hour measurement, when the relative positional relationship between the camera and the worker changes, etc., even though the worker is working, it may be determined that the area where the work object is present does not correspond to the area where the worker is present. In this case, the working hours will not be accurately measured. That is, there is room for improvement in the accuracy of working hour measurement.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to improve the accuracy of working hour measurement.

Means for Solving the Problems

[0005] The information processing apparatus according to the embodiment includes an extraction unit, an acquisition unit, an association unit, and a measurement unit. The extraction unit extracts in-work images during work in which a series of multiple work types of work are performed from an image continuously capturing the actions of a worker. The acquisition unit acquires time-series information representing the characteristics of the actions of the worker at each time point in time-series order from each of the in-work images. The association unit compares the known model time-series information of the work type of the work and the switching time of the work with the time-series information, and associates the time points at which the characteristics of the actions are similar. The measurement unit measures the working hours for each work type for the time-series information based on the association result. The acquisition unit extracts feature vectors from the work-in-progress image, calculates the relative distance between multiple feature locations on the work-in-progress image corresponding to the feature elements of the extracted feature vectors, and obtains time-series information by arranging the feature vectors, with the vectors representing the calculated relative distances added as new feature elements, in chronological order. The multiple feature locations include the position of at least one tool and the position of the worker's hand. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a block diagram showing an example of the configuration of a work time measurement system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the CPU of a work time measuring device according to an embodiment. [Figure 3] Figure 3 illustrates an example of the process for acquiring work video according to this embodiment. [Figure 4] Figure 4 illustrates an example of the time-series data acquisition process according to the embodiment. [Figure 5] Figure 5 illustrates an example of time-series data storage processing according to this embodiment. [Figure 6] Figure 6 illustrates an example of the selection process for target data and example data according to the embodiment. [Figure 7] Figure 7 illustrates an example of the sample data extraction process according to the embodiment. [Figure 8] Figure 8 illustrates an example of the correspondence process between target data and example data according to the embodiment. [Figure 9] Figure 9 illustrates an example of the estimation process for work types and the timing of work type changes according to the embodiment. [Figure 10] Figure 10 is a diagram illustrating an example of the work time measurement process according to the embodiment. [Figure 11] Figure 11 illustrates an example of a process performed by a work time measuring device according to an embodiment. [Figure 12] Figure 12 illustrates an example of the process performed by the work time measuring device according to Modification Example 1. [Figure 13] Figure 13 illustrates an example of the process performed by the work time measuring device according to Modification 2. [Figure 14] Figure 14 illustrates an example of the process performed by the work time measuring device according to Modification 3. [Modes for carrying out the invention]

[0007] Embodiments of the information processing device and program will be described in detail below with reference to the attached drawings. In the following embodiments, an example will be described in which the information processing device and program of the present invention are applied to measuring the working time of workers performing various tasks on a production line, but the present invention is not limited to this embodiment.

[0008] Figure 1 is a block diagram showing an example of the configuration of the information processing system S according to this embodiment. The work management system 1 includes a work time measuring device 10, a work information server 20, and a network camera 30. The work time measuring device 10, the work information server 20, and the network camera 30 are connected via a network such as the Internet or a LAN (Local Area Network).

[0009] The work information server 20 is a server device that manages information related to the work of workers. For example, it manages information related to the work time of each worker. In addition, for example, the work information server 20 stores information related to the standard work time for each type of work.

[0010] The network camera 30 is a camera that can be connected to a network such as the internet or a LAN. Multiple network cameras 30 may be installed. The network camera 30 is installed in a position that can photograph workers performing tasks on the production line.

[0011] For example, the network camera 30 captures the work of workers on the production line. By using the network camera 30, it is possible to capture the work of workers who are working in locations far from the installation location of the work time measurement device 10, which will be described later. The time at which the image was taken is recorded in the image (video) captured by the network camera 30.

[0012] The working time measurement device 10 is a device that measures the working time of workers. For example, the working time measurement device 10 measures the working time of workers on a production line. Hereinafter, the configuration of the working time measurement device 10 will be described with reference to FIGS. 2 and 3.

[0013] First, the hardware configuration of the working time measurement device 10 will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the working time measurement device 10.

[0014] As shown in FIG. 2, the working time measurement device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a communication I / F (Interface) 14, a camera I / F 15, and a storage unit 16. These components 11 to 16 are connected to each other via a data bus.

[0015] In addition, the working time measurement device 10 is connected to a camera 17 via the camera I / F 15. The camera 17 incorporates an RTC (Real Time Clock) 18. The RTC 18 measures the current date and time.

[0016] The CPU 11 controls the working time measurement device 10 overall. For example, the CPU 11 uses the RAM 12 as a working area and executes various processes by executing programs stored in the ROM 13, the storage unit 16, etc. The storage unit 16 is a non-volatile memory capable of writing and rewriting data. For example, the storage unit 16 stores target data 161 and template data 162 described later.

[0017] The communication I / F 14 is a device for establishing communication in a predetermined format with the work information server 20 and the network camera 30. The communication format is not particularly limited, and examples include those conforming to standards such as Ethernet (registered trademark), IEEE802.11, etc.

[0018] The camera I / F 15 controls the camera 17 according to the instructions of the CPU 11, captures the image signal output from the camera 17, and outputs it as a captured image. The camera 17 performs imaging according to the control of the camera I / F 15 and outputs an image signal. The number of cameras 17 connected to the working time measurement device 10 may be one or more. The camera 17 is provided at a position where it can photograph an operator working on the production line.

[0019] For example, the camera 17 photographs the working state of an operator working on the production line. The imaging time of the image (moving image) captured by the camera 17 is recorded using the RTC 18.

[0020] Also, the working time measurement device 10 includes functional units realized by the cooperation of the CPU 11 and programs stored in the ROM 13 and the storage unit 16. Specifically, as shown in FIG. 2, when the CPU 11 executes a program, functional units such as the first acquisition unit 111, the detection unit 112, the generation unit 113, the second acquisition unit 114, the storage control unit 115, the first selection unit 116, the second selection unit 117, the association unit 118, the estimation unit 119, and the measurement unit 120 are realized on the RAM 12. In addition, target data 161 and template data 162 are stored in the storage unit 16.

[0021] The first acquisition unit 111 acquires a plurality of images of the work performed by the operator. For example, the first acquisition unit 111 acquires the moving image captured by the camera 17. Also, for example, the first acquisition unit 111 acquires the moving image captured by the network camera 30. Note that the first acquisition unit 111 may acquire only one of the moving images captured by the camera 17 and the network camera 30.

[0022] The detection unit 112 detects a work start trigger or a work end trigger. For example, the detection unit 112 detects a work start trigger or a work end trigger from the moving image acquired by the first acquisition unit 111. The work start trigger is, for example, a specific posture taken by the operator when starting work. The work end trigger is, for example, a specific posture taken by the operator when ending work.

[0023] In this embodiment, the work start trigger is described as the worker assuming a specific posture, but the work start trigger is not limited to this. For example, the work start trigger may be the detection of the worker pressing a work start input button provided on the work time measuring device 10.

[0024] Furthermore, the work start trigger may be, for example, detecting an audio signal indicating the start of work, when the camera 17 or network camera 30 records video and audio simultaneously. Alternatively, the work start trigger may be, for example, detecting the elapsed time of a predetermined work start date.

[0025] Furthermore, in this embodiment, the work completion trigger is described as the worker adopting a specific posture different from the work start trigger, but the work completion trigger is not limited to this. The work completion trigger may, for example, be the detection of the worker pressing a work completion input button provided on the work time measuring device 10.

[0026] Furthermore, the work completion trigger may be, for example, the detection of audio indicating the completion of work by recording video and audio simultaneously with video using camera 17 or network camera 30. Alternatively, the work completion trigger may be, for example, the detection of the elapsed time of a predetermined work completion period.

[0027] Furthermore, for example, if the detection unit 112 detects a work start trigger from the video acquired by the first acquisition unit, it records the work start time in the video. Similarly, if the detection unit 112 detects a work end trigger from the video acquired by the first acquisition unit, it records the work end time in the video.

[0028] The generation unit 113 generates a one-cycle work video (hereinafter referred to as a one-cycle video) by extracting video footage of a series of multiple work types performed during work from video footage of the worker's movements captured by the camera 17 or network camera 30. The generation unit 113 is an example of an extraction unit. The one-cycle video is an example of a video of work in progress.

[0029] For example, if the detection unit 112 has recorded both the start time and end time of the work in the video, the generation unit 113 generates a one-cycle video based on the start time and end time of the work. Below, an example of the one-cycle video generation process will be explained using Figure 3. Figure 3 is a diagram illustrating an example of the one-cycle video generation process.

[0030] In Figure 3, at time t+0, the detection unit 112 detects the start of work, and the start time of work is recorded. Similarly, at time t+3, the detection unit 112 detects the end of work, and the end time of work is recorded. The generation unit 113 generates a 1-cycle video Pa from the start time of work (t+1) to the start time of work (t+2).

[0031] The one-cycle video Pa in Figure 3 records that the worker is performing task A at time t+2 and performing task B at time t+3. Task A and task B represent different types of tasks. In Figure 3, the worker is performing two types of tasks, task A and task B, but the number of task types is not limited to two. For example, there may be three or more types of tasks.

[0032] Furthermore, in Figure 3, at time t+5, the detection unit 112 detects the start of work, and the start time of work is recorded. Similarly, at time t+9, the detection unit 112 detects the end of work, and the end time of work is recorded. The generation unit 113 generates a 1-cycle video Pb from the start time of work (t+6) to the start time of work (t+8).

[0033] The one-cycle video Pb in Figure 3 records that the worker was performing task A at time t+6, and was performing task B at times t+7 and t+8.

[0034] Furthermore, the video footage acquired by the first acquisition unit 111 also includes scenes (times t+4, t+10) where the worker is performing activities related to the work that are not included in the work itself, but these are not included in the 1-cycle video footage Pa and Pb. In this way, the generation unit 113 can generate video footage that records only the scenes where the worker is performing activities that are included in the work.

[0035] Returning to Figure 2, let's continue the explanation. The second acquisition unit 114 acquires time-series data that represents the characteristics of the worker's actions at each point in time in chronological order. Time-series data is an example of time-series information.

[0036] For example, the second acquisition unit 114 extracts feature vectors for each frame by applying feature extraction to each image that makes up one cycle of motion. The second acquisition unit 114 also acquires time-series data by arranging the extracted feature vectors for each frame in the time direction. Known techniques such as human action recognition, skeletal detection, object detection, and vectorization of the image itself can be used as appropriate for feature extraction.

[0037] The second acquisition unit 114 may pre-store images of workers' faces and identify workers appearing in the video using known image recognition technology. In this case, the second acquisition unit 114 may associate information identifying the identified worker with time-series data.

[0038] The process of acquiring time-series data will be explained below using Figure 4. Figure 4 is a diagram illustrating an example of the process of acquiring time-series data.

[0039] The one-cycle video Px in Figure 4 contains frames Fa, which was captured at time t+1, and Fk, which was captured at time t+9. The second acquisition unit 114 applies feature extraction to frames Fa to Fk, respectively, and extracts feature vectors Ba to Bk from each of frames Fa to Fk. Next, the second acquisition unit 114 obtains time-series data Dx by arranging the feature vectors Ba to Bk in chronological order.

[0040] In Figure 4, it is assumed that the worker is performing task A in frames Fa~Fc (times t+0~t+2) and Fg (time t+6). Also, in frames Fd~Ff (times t+3~t+5), the worker is assumed to be away from the workspace (away from their seat). Furthermore, in frames Fi~Fk (times t+7~t+10), the worker is assumed to be performing task B.

[0041] It is likely that at least one of the following elements differs between Task A, leaving the desk, and Task B in the captured images: the worker's position, the worker's posture, the position of the object being worked on, the angle of the object, etc. From this, it is likely that the feature vectors Ba~Bc, Bg, Be, Bf, and Bh~Bk have different properties. Therefore, the time-series data Dx in Figure 4 can be said to represent the changes in image features associated with changes in the worker's activity from time t+0 to t+11 in a time-series format.

[0042] Returning to Figure 2, let's continue the explanation. The memory control unit 115 controls the time-series data acquired by the second acquisition unit 114 to be stored in either the target data 161 or the example data 162 of the memory unit 16. The target data 161 is an example of the first group. The example data 162 is an example of the second group.

[0043] Here, target data 161 stores multiple target time series data. Target time series data are, for example, time series data that are used to estimate work type and work changeover time. Target time series data is an example of target time series information.

[0044] Furthermore, the example data 162 stores multiple example time series data. For example, the example time series data is time series data in which the type of work and the work changeover time are known. Specifically, the example time series data is time series data in which the breakdown of work types is fixed, such as performing work A from time t+a to t+c, switching from work A to work B at t+d, and performing work B from time t+d to t+e. The example time series data is an example of example time series information.

[0045] The time-series data storage process will be explained below using Figure 5. Figure 5 is a diagram illustrating an example of the time-series data storage process. In Figure 5, the time-series data Dx represents specific time-series data acquired by the second acquisition unit 114. As an example, if the time-series data Dx is time-series data selected as example time-series data by a user, such as the administrator of the work time measurement device 10, the storage control unit 115 controls the storage of the example data 162 in the storage unit 16.

[0046] Furthermore, if the time-series data Dx is time-series data that was not selected by the user as example time-series data, the memory control unit 115 controls the storage unit 16 to store it in the target data 161.

[0047] In this embodiment, the memory control unit 115 controls the time-series data to be stored in either the target data 161 or the example data 162 according to the user's selection. However, the method for storing the time-series data in either the target data 161 or the example data 162 is not limited to the above.

[0048] For example, the memory control unit 115 may automatically store time-series data recording the work performed by each of multiple workers from the beginning of the day to a predetermined number of times in the example data 162. Alternatively, for example, the memory control unit 115 may automatically store time-series data recording the work of a specific worker in the example data 162.

[0049] In this embodiment, the user inputs a breakdown of the types of work (hereinafter also referred to as teaching data) for the example time series data stored in the example data 162. For example, the user inputs that work A is performed from time t+a to t+c, that work A is switched to work B at t+d, and that work B is performed from time t+d to t+e for the example time series data stored in the example data 162. The memory control unit 115 records the input content in the example time series data.

[0050] The method for acquiring teaching data is not limited to the above. For example, standard work data, which records standard work for each type of work, may be prepared, and teaching data may be acquired by identifying the type of work being performed by the worker in the video based on the similarity between the feature vectors of the standard work data and the feature vectors of the example time series data.

[0051] Returning to Figure 2, let's continue the explanation. The first selection unit 116 selects one time series data from the target time series data stored in the target data 161. For example, the first selection unit 116 selects the target time series data stored in the target data 161 that has the earliest recorded work start time.

[0052] The second selection unit 117, when a target time series data is selected, selects one example time series data from among multiple example time series data stored in the example data 162. For example, the second selection unit 117 first selects multiple example time series data from among multiple example time series data stored in the example data 162.

[0053] For example, the second selection unit 117 refers to the start times of work recorded in multiple example time series data and selects a predetermined number of example time series data in order of proximity to the start times of work recorded in the target time series data selected by the first selection unit 116. Note that the criteria for the multiple example time series data are not limited to the above. For example, the second selection unit 117 may randomly select a predetermined number of example time series data.

[0054] Next, the second selection unit 117 selects the example time series data from among the selected example time series data that has the lowest mapping cost when mapped to the target time series data selected in the first selection unit 116. Note that known methods such as DTW (Dynamic Time Warping) can be used as appropriate for mapping the target time series data to the example time series data.

[0055] The time series data selection process will be explained below using Figures 6 and 7. Figures 6 and 7 illustrate an example of the time series data selection process. As shown in the upper part of Figure 6, the first selection unit 116 selects the target time series data ODa from the target time series data stored in the target data 161.

[0056] Furthermore, as shown in the lower part of Figure 6, the second selection unit 117 selects the model time series data EDa and model time series data EDb from the model time series data stored in the model data 162. In this example, the second selection unit 117 selects two model time series data, but the second selection unit 117 may select three or more model time series data.

[0057] As shown in the lower part of Figure 6, the model time series data EDa and model time series data EDb consist of records of the period during which the worker is performing task A (indicated as A in Figure 6), records of the period during which the worker is performing task B (indicated as B in Figure 6), and records of the period during which the worker is performing task C (indicated as C in Figure 6).

[0058] Furthermore, the model time series data EDa and EDb contain the task changeover time CPa, when the worker switches from task A to task B, and the task changeover time CPb, when the worker switches from task B to task C, as teaching data.

[0059] Furthermore, the second selection unit 117 may exclude from selection any example time-series data stored in the example data 162 that does not have instruction data recorded on it. Also, if the second selection unit 117 selects example time-series data that does not have instruction data recorded on it, it may prompt the user to input the instruction data.

[0060] Next, the second selection unit 117 calculates the distance between the target time series data ODa and the example time series data EDa and the example time series data EDb, respectively, as shown in Figure 7. In Figure 7, the distance between the target time series data ODa and the example time series data EDa is 58, and the distance between the target time series data ODa and the example time series data EDb is 100.

[0061] Here, a shorter distance results in a lower mapping cost by DTW. Therefore, in this embodiment, the second selection unit 117 extracts the example time series data with the smallest distance from among the multiple selected example time series data. In the example in Figure 7, the mapping cost of example time series data EDa to the target time series data ODa is lower than that of example time series data EDb. Therefore, the second selection unit 117 extracts (selects) example time series data EDa.

[0062] Returning to Figure 2, let's continue the explanation. The correspondence unit 118 compares the example time series information with the target time series information and associates points in time when the characteristics of their operation are similar. For example, the correspondence unit 118 uses a known method such as DTW to associate one target time series data selected by the first selection unit 116 with one example time series data selected by the second selection unit 117 at the frame level.

[0063] The mapping process will be explained below using Figure 8. Figure 8 is a diagram illustrating an example of the mapping process. As a prerequisite, it is assumed that the target time series data ODa has been selected in the first selection unit 116 and extracted (selected) in the second selection unit 117.

[0064] In the example in Figure 8, one frame is represented by one circle. That is, the target time series data ODa consists of 13 frames. On the other hand, the example time series data EDa consists of 11 frames. The mapping unit 118 uses DTW to map the target time series data ODa and the example time series data EDa at the frame level. As shown in Figure 8, the mapping unit 118 can map time series data with different numbers of frames.

[0065] Returning to Figure 2, let's continue the explanation. The estimation unit 119 estimates the work period and work changeover time for each type of work in the target time series data based on the correspondence results from the correspondence unit 118. The estimation process will now be explained using Figure 9. Figure 9 is a diagram illustrating an example of the estimation process. As a premise, the first circle in the target time series data ODa and the example time series data EDa represents the frame corresponding to the work start time, and the last circle represents the frame corresponding to the work end time.

[0066] First, the estimation unit 119 estimates the frame shown in (1) corresponding to the work changeover time CPa as the estimated changeover time EPa. Similarly, it estimates the frame shown in (2) corresponding to the work changeover time CPb as the estimated changeover time EPb. If there are multiple frames corresponding to the work changeover time, the midpoint between the first and last frame among the corresponding frames may be estimated as the estimated changeover time.

[0067] Next, as shown in Figure 9, the estimation unit 119 estimates the period from the start time of work to the estimated switching time EPa as the estimated period EA, which is estimated to be the implementation period of work A. The estimation unit 119 also estimates the period from the estimated switching time EPa to the estimated switching time EPb as the estimated period EB, which is estimated to be the implementation period of work B. Furthermore, the estimation unit 119 estimates the period from the estimated switching time EPb to the end time of work as the estimated period EC, which is estimated to be the implementation period of work C.

[0068] Returning to Figure 2, the explanation continues. The measurement unit 120 measures the work time for each type of work based on the correspondence results from the correspondence unit 118. The measurement unit 120 also transmits the measurement results to the work information server 20 via the communication interface. The measurement process will now be explained using Figure 10. Figure 10 is a diagram illustrating an example of the measurement process.

[0069] As shown in Figure 10, the measurement unit 120 measures the lengths of the estimated period EA, estimated period EB, and estimated period EC estimated by the estimation unit 119 as the work time for work A, work time for work B, and work time for work C, respectively. In the example in Figure 10, the measurement unit 120 measures that the work time for work A is 15.5 seconds, the work time for work B is 54.2 seconds, and the work time for work C is 8.9 seconds.

[0070] Next, we will explain the processes performed by the work time measurement device 10. Figure 11 is a flowchart showing an example of the processes performed by the work time measurement device 10.

[0071] First, the first acquisition unit 111 acquires camera images that record the work performed by the worker (step S101). For example, the first acquisition unit 111 acquires video from at least one of the camera 17 and the network camera 30. The video contains the time when the video was taken.

[0072] Next, the detection unit 112 determines whether it has detected a work start trigger from the video image acquired in step S101 (step S102). If it has not detected a work start trigger (step S102: No), it proceeds to step S104.

[0073] If a work start trigger is detected (step S102: Yes), the detection unit 112 records the time at which the work start trigger was detected on the video acquired in step S101 as the work start time on the video (step S103).

[0074] Next, the detection unit 112 determines whether it has detected a work completion trigger from the video image acquired in step S101 (step S104). If a work completion trigger has not been detected (step S104: No), the process returns to step S101.

[0075] If a work completion trigger is detected (step S104: Yes), the detection unit 112 records the time at which the work completion trigger was detected on the video acquired in step S101 as the work completion time on the video (step S105).

[0076] Next, the generation unit 113 determines whether both the work start time and the work end time are recorded in the video acquired in step S101 (step S106). If at least one of the work start time and work end time is not recorded (step S106: No), the process returns to step S101. If both the work start time and work end time are recorded (step S106: Yes), the generation unit 113 generates a 1-cycle video (step S107).

[0077] Next, the second acquisition unit 114 acquires time-series data based on the one-cycle video (step S108). For example, the second acquisition unit 114 extracts feature vectors for each frame from the one-cycle video and acquires time-series data by arranging the extracted feature vectors for each frame in the time direction.

[0078] Next, the memory control unit 115 adds the time-series data acquired in step S108 to the target data 161 or the example data 162 (step S109). For example, the memory control unit 115 stores the time-series data selected by the user as example time-series data in the example data 162. The memory control unit 115 also stores time-series data that has not been selected as example time-series data in the target data 161.

[0079] Next, the memory control unit 115 determines whether time-series data has been added to the example data 162 (step S110). If time-series data has not been added to the example data 162, in other words, if time-series data has been added to the target data 161 (step S110: No), the process proceeds to step S112.

[0080] On the other hand, if time-series data is added to the example data 162 (step S110: Yes), the memory control unit 115 acquires teaching data (step S111). For example, the memory control unit 115 receives input of teaching data from the user and acquires the input content as teaching data.

[0081] Next, the first selection unit 116 determines whether there is one or more time-series data in the target data 161 and two or more time-series data in the example data (step S112). If no time-series data exists (step S112: No), the process returns to step S101.

[0082] On the other hand, if time series data exists (step S112: Yes), the first selection unit 116 selects one target time series data from the target data 161 (step S113). For example, the first selection unit 116 selects the target time series data that records the earliest time of work among the target time series data stored in the target data 161.

[0083] Next, the second selection unit 117 selects multiple target time series data from the example data 162 (step S114). For example, the second selection unit 117 refers to the times recorded in multiple example time series data stored in the example data 162 and selects a predetermined number of example time series data in order of the time recorded being closest to the time recorded in the target time series data selected in step S113.

[0084] Next, the second selection unit 117 calculates the distance between the one target time series data selected in step S113 and the multiple example time series data selected in step S114 (step S115). The second selection unit 117 then extracts the example time series data from the multiple example time series data that has the smallest distance calculated in step S115 (step S116).

[0085] Next, the matching unit 118 associates the target time series data selected in step S113 with the example time series data selected in step S116 (step S117). For example, the matching unit 118 uses DTW to associate the target time series data and the example time series data at the frame level.

[0086] Next, the estimation unit 119 estimates the work changeover time and work type of the target time series data (step S118). For example, the estimation unit 119 estimates the work changeover time and work type for each frame based on the correspondence result of step S117. Next, the measurement unit 120 measures the work time (step S119). For example, the measurement unit 120 measures the work time for each work type based on the estimation result of step S118.

[0087] Next, the first acquisition unit 111 determines whether it has received a termination instruction from the user to end the process (step S120). If no termination instruction has been received (step S120: No), the process returns to step S101. If a termination instruction has been received (step S120: Yes), this process is terminated.

[0088] As described above, the work time measurement device 10 according to this embodiment acquires time-series data based on multiple images recording multiple tasks of multiple work types performed by a worker, selects one target time-series data from multiple target time-series data to be measured for work time, selects one example time-series data from multiple example time-series data with known work times, associates the two selected data, and measures the work time for each work type for the target time-series data based on the association result.

[0089] As described above, the work time measurement device 10 according to this embodiment associates the target time series data with the example time series data and measures the work time for each type of work from the matching result. Therefore, even if the field of view of the camera that captured the video image that forms the basis of the time series data changes, it is possible to associate the target time series data with the example time series data and estimate the work time for each type of work of the target time series data. Accordingly, the work time measurement device 10 according to this embodiment can measure the work time for each type of work with high accuracy even in such cases. In other words, the work time measurement device 10 according to this embodiment can improve the accuracy of work time measurement.

[0090] The above-described embodiments can also be modified and implemented as appropriate by changing the configuration or some of the functions of each device in the information processing system S. Therefore, several modifications of the above-described embodiments will be described below as other embodiments. In the following, we will mainly describe the differences from the above-described embodiments, and will omit detailed explanations of points that are common with what has already been described. Furthermore, the modifications described below may be implemented individually or in combination as appropriate.

[0091] (Variation 1) In the above-described embodiment, a method was explained in which feature vectors for each frame are extracted using techniques such as human action recognition, skeleton detection, object detection, and vectorization of the image itself. In this modified example, a method is described in which the relative distance is calculated for each position of features that can be represented by coordinates on the image, such as the position of the worker's skeleton and the position of objects, and the calculated relative distance is added to the feature vector as a new feature element.

[0092] In this modified example, the second acquisition unit 114 calculates the relative distance between multiple feature points, such as the position of the worker's skeleton and the position of objects, on each image (frame) that constitutes the one-cycle video generated by the generation unit 113. The second acquisition unit 114 adds the calculated relative distance vector as a new feature element to the feature vector for each frame extracted by applying feature extraction. The second acquisition unit 114 acquires time-series data by arranging the feature vectors with the newly added feature elements in chronological order.

[0093] The following describes the processes performed by the work time measuring device 10 according to this modified example. Figure 12 is a flowchart showing an example of the processes performed by the work time measuring device 10 according to Modified Example 1. Steps S201 to S207 are the same as steps S101 to S107 in Figure 11, so their explanation is omitted.

[0094] After step S207, the second acquisition unit 114 applies feature extraction to each image constituting the one-cycle video generated in step S207 to extract feature vectors for each frame (step S208). Next, the second acquisition unit 114 calculates the relative distance between feature points, which are feature elements of the extracted feature vectors, such as the position of the worker's skeleton and the position of objects on each image constituting the one-cycle video (step S209).

[0095] Next, the second acquisition unit 114 acquires time-series data (step S210). For example, the second acquisition unit 114 adds a vector representing the relative distance calculated in step S209 as a new feature element to the extracted feature vector, and acquires time-series data by arranging the feature vectors to which the new feature element has been added in chronological order. Steps S211 to S222 are the same as steps S109 to S120 in Figure 11, so their explanation is omitted.

[0096] In this modified version, the time-series data includes information representing the relative distance between feature points. Therefore, the work time measurement device 10 in this modified version can, for example, acquire location information of tools with a fixed usage location, such as a screwdriver, installed in the workspace. This allows it to determine that the screwdriver has been used even if the worker's position in the image changes slightly, as long as the relative distance between the worker's hand and the screwdriver decreases. As a result, the work time measurement device 10 in this modified version can estimate the period during which the worker is performing a task, and can also estimate the task changeover time and each type of task.

[0097] (Modification 2) In the above-described embodiment, a configuration was described in which the second selection unit 117 selects multiple example time series data from the example time series data 162 based on the time recorded in the target time series data and the time recorded in the example time series data. In this modified example, a configuration is described in which multiple example time series data are selected from the example time series data 162 using information related to the work.

[0098] In this modified version, the one-cycle video generated by the generation unit 113 is stored in the work information server 20. Furthermore, work information, which is information related to the work, is associated with the one-cycle video.

[0099] Work information includes, for example, a product ID that identifies the product related to the work performed, a worker ID that identifies the worker who performed the work, the worker's length of service, the product ID of the product the worker has experience with, and work speed for each type of work. As an example, the work information server 20 receives input of work information from the administrator of the work information server 20, etc., and stores it in the storage device provided by the work information server 20 in association with a 1-cycle video.

[0100] The work information server 20 may also store information for each worker in advance, such as a facial image, worker ID, length of service, product ID of products the worker has experience with, and work speed for each type of work, and acquire work information based on this information.

[0101] For example, the work information server may use known image recognition technology to extract the face image of the worker shown in the one-cycle video from the one-cycle video, and then acquire work information based on the information corresponding to that face image. In this case, by pre-maintaining information that associates product images with product IDs, the product involved in the work can also be identified from the one-cycle video.

[0102] In this modified example, the second acquisition unit 114 acquires work information from the work information server 20 and processes the acquisition of work information to link it to the acquired time-series data. The second selection unit 117 compares the work information of the target time-series data selected by the first selection unit 116 with the work information of multiple example time-series data stored in the example data 162. The second selection unit 117 then selects a predetermined number of example time-series data in descending order of similarity of work information (degree of agreement of information contained in the work information).

[0103] The following describes the processes performed by the work time measuring device 10 according to this modified example. Figure 13 is a flowchart showing an example of the processes performed by the work time measuring device 10 according to Modified Example 2. Steps S301 to S307 are the same as steps S101 to S107 in Figure 11, so their explanation is omitted. However, in this modified example, the 1-cycle video generated in step S307 is stored in the work information server 20.

[0104] After step S307, the second acquisition unit 114 acquires work information to be linked to the time-series data (step S308). For example, the second acquisition unit 114 acquires work information associated with a one-cycle video stored in the work information server 20 as work information to be linked to the time-series data.

[0105] Step S309 is the same as step S108 in Figure 11, but in this modified example, the second acquisition unit 114 acquires time-series data from a one-cycle video corresponding to the work information acquired in step S308. The second acquisition unit 114 then associates the work information acquired in step S308 with the time-series data. Steps S310 to S314 are the same as steps S109 to S113 in Figure 11, so their explanation is omitted.

[0106] After step S314, the second selection unit 117 calculates the similarity of each of the multiple example time series data stored in the example data 162 to the target time series data selected in step S314 (step S315).

[0107] Next, the second selection unit 117 selects multiple example time series data based on the similarity calculated in step S315 (step S316). For example, the second selection unit 117 selects a predetermined number of example time series data in descending order of similarity to the target time series data. Steps S317 to S322 are the same as steps S115 to S120 in Figure 11, so their explanation is omitted.

[0108] According to this modified version, when selecting multiple example time series data from example time series data stored in example data 162, it is possible to select example time series data that is similar in work information to the selected target time series data. For example, if the worker related to the target time series data and the worker related to the example time series data are the same, there is a high probability that the characteristics of both will be similar. Therefore, the work time measurement device 10 according to this modified version can increase the probability of selecting example time series data that has similar characteristics to the target time series data by selecting example time series data that has similar work information to the target time series data. Furthermore, by selecting example time series data that has similar characteristics to the target time series data, it is possible to improve the accuracy of estimating the work changeover time and work type, and thus the work time measurement device 10 according to this modified version can improve the accuracy of work time measurement.

[0109] (Variation 3) In the above embodiment, a method was described in which time-series data is added to the model data 162 according to the selection of an administrator or the like. In this modified example, a method is described in which target time-series data added to target data 161 is added to the model data 162 when certain conditions are met.

[0110] In this modified example, the memory control unit 115 performs control to move a target time series data from target data 161 to model data 162 when the automatic addition setting for example time series data has been set by an administrator or the like, and a certain target time series data satisfies predetermined conditions.

[0111] For example, if the automatic addition setting for example time series data is ON, and the difference in time length between the target time series data and the standard time stored in the work information server 20 is less than a threshold, the memory control unit 115 moves the target time series data from the target data 161 to the example data 162.

[0112] In this embodiment, the standard time is information representing the standard working time for each type of work. The standard time is defined for each work process, such as assembling product A and assembling product B.

[0113] Furthermore, for example, if the automatic addition setting for example time series data is ON and the distance to all example time series data stored in example data 162 is less than a threshold, the memory control unit 115 moves the target time series data from target data 161 to example data 162. The memory control unit 115 may also move the target time series data from target data 161 to example data 162 if at least one of the time difference with the standard time and the distance to the example time series data is less than a threshold.

[0114] The following describes the processes performed by the work time measuring device 10 according to this modified example. Figure 14 is a flowchart showing an example of the processes performed by the work time measuring device 10 according to Modified Example 3. Steps S401 to S419 are the same as steps S101 to S119 in Figure 11, so their explanation is omitted.

[0115] After step S419, the memory control unit 115 determines whether the automatic addition setting for example time series data is turned ON (step S420). If the automatic addition setting is not turned ON (step S420: No), the process proceeds to step S424. If the automatic addition setting is turned ON (step S420: Yes), the memory control unit 115 obtains the standard time for the work process related to the target time series data from the work server (step S421).

[0116] Next, the memory control unit 115 determines whether at least one of the time difference with the standard time and the distance with the example time series data is less than a threshold (step S422). If both are greater than or equal to the threshold (step S422: No), the process proceeds to step S424.

[0117] On the other hand, if at least one of the values ​​is below the threshold (step S422: Yes), the target time series data for which the work time was measured in step S419 is moved from target data 161 to example data 162 (step S423). Step S424 is the same as step S120 in Figure 11, so the explanation is omitted.

[0118] This modified version allows for the use of a larger amount of time series data as model time series data. This increases the likelihood of selecting model time series data with a high degree of similarity to the target time series data when selecting model time series data. In other words, the work time measurement device 10 according to this modified version can improve the accuracy of work time measurement.

[0119] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0120] 10. Work time measuring device 11 CPU 12 RAM 13 ROM 14 Communication I / F 15 Camera I / F 16 Memory section 17 Cameras 18 RTC 20. Work Information Server 30 Network Cameras 111 First acquisition part 112 Detection unit 113 Generation part 114 Second Acquisition Department 115 Memory Control Unit 116 First Selection Section 117 Second Selection Section 118 Correspondence section 119 Estimation Department 120 Measurement section [Prior art documents] [Patent Documents]

[0121] [Patent Document 1] Japanese Patent Publication No. 2020-135767

Claims

1. An extraction unit extracts images of a worker performing a series of multiple types of tasks from a series of images capturing the worker's movements. An acquisition unit that acquires time-series information representing the characteristics of the worker's actions at each point in time, in chronological order, from each of the aforementioned work images, A matching unit compares the type of work and the timing of the switchover of the work with known example time-series information and associates points in time when the characteristics of the operation are similar. Based on the correspondence results of the aforementioned correspondence unit, a measurement unit measures the work time for each type of work with respect to the time-series information, Equipped with, The acquisition unit extracts feature vectors from the working image, calculates the relative distances between multiple feature locations on the working image corresponding to the feature elements of the extracted feature vectors, and acquires the time-series information by arranging the feature vectors, with the vectors representing the calculated relative distances added as new feature elements, in chronological order. The aforementioned plurality of feature locations include the location of at least one tool and the location of the worker's hand. Information processing device.

2. A first selection unit selects one of the target time series information from among the acquired time series information that is used to estimate the work type of multiple tasks and the switching time of those tasks. The system further comprises a second selection unit that selects one example time series from a plurality of example time series information, The matching unit associates the selected target time series information with the selected example time series information at points in time when the characteristics of the operation are similar. The measurement unit measures the work time for each type of work for the target time-series information based on the correspondence results. The information processing apparatus according to claim 1.

3. The acquisition unit associates the target time-series information and the model time-series information with work information that includes at least one of the following: worker identification information related to the target time-series information or model time-series information, worker work experience, worker work time, and product identification information related to the time-series information. The second selection unit selects the example time series information based on the similarity between the target time series information selected by the first selection unit and the work information associated with it. The information processing apparatus according to claim 2.

4. The memory control unit further comprises a memory unit that determines whether the time series information is the target time series information or the example time series information, and stores the time series information determined to be the target time series information as a first group and the time series information determined to be the example time series information as a second group in the memory unit. The first selection unit selects one of the target time-series information from the first group, The second selection unit, when a plurality of example time series information is stored as the second group, compares the target time series information selected by the first selection unit with the plurality of example time series information and selects the one example time series information that minimizes the matching cost. The information processing apparatus according to claim 2.

5. The memory control unit, if it satisfies at least one of the following conditions, that the cost of associating the target time series information with the example time series information stored in the memory device as the second group is less than a threshold, and that the difference in time length between the measured work time for each work type and the standard time for each work type is less than a threshold, then controls the target time series information to be used as the example time series information and move it from the first group to the second group. The information processing apparatus according to claim 4.

6. Computers, An extraction unit extracts images of a worker performing a series of multiple types of tasks from a series of images capturing the worker's movements. An acquisition unit that acquires time-series information representing the characteristics of the worker's actions at each point in time, in chronological order, from each of the aforementioned work images, A matching unit compares the type of work and the timing of the switchover of the work with known example time-series information and associates points in time when the characteristics of the operation are similar. Based on the correspondence results of the aforementioned correspondence unit, a measurement unit measures the work time for each type of work with respect to the time-series information, It is a program that makes it function as such. The acquisition unit extracts feature vectors from the working image, calculates the relative distances between multiple feature locations on the working image corresponding to the feature elements of the extracted feature vectors, and acquires the time-series information by arranging the feature vectors, with the vectors representing the calculated relative distances added as new feature elements, in chronological order. The aforementioned plurality of feature locations include the location of at least one tool and the location of the worker's hand. program.

Images