Work analysis device and method

The work analysis device efficiently identifies and extracts abnormal video segments from workplace footage, addressing the inefficiency of manual review in existing systems by automating the detection of abnormal conditions in production processes.

WO2024247644A1PCT designated stage expired Publication Date: 2024-12-05PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017265
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-09
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing work management systems require line managers to manually review extensive video footage to identify abnormal conditions in production processes, which is inefficient and time-consuming.

Method used

A work analysis device and method that analyzes video captured by cameras at a workplace, detecting abnormal periods in task progress and extracting relevant video segments showing the affected area and surrounding processes, allowing for efficient identification of issues.

Benefits of technology

Enables line managers to quickly analyze work by viewing only the abnormal video segments, reducing processing time and improving management efficiency by highlighting specific issues within the production process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017265_05122024_PF_FP_ABST
    Figure JP2024017265_05122024_PF_FP_ABST
Patent Text Reader

Abstract

This work analysis device is for analyzing a video obtained by imaging, with one or more cameras, a workplace in which a plurality of tasks are performed, said device comprising: a storage unit that stores video data indicating a captured video in which is captured a work area in which each task is performed at a workplace; and a control unit that detects, on the basis of work information indicating the progress state of a plurality of tasks, an abnormal period in which the progress state deviates from a predetermined reference related to each task among the plurality of tasks. On the basis of the detected abnormal period, the control unit extracts, from a captured video indicated by the video data, a first video in which is captured a first work area including a location where a first work task was performed during the abnormal period, and a second video related to the first video. The second video includes an imaging result of a second work area located in the vicinity of the first work area and / or an imaging result of a second task performed before and / or after the first task.
Need to check novelty before this filing date? Find Prior Art

Description

Work analysis device and method

[0001] The present disclosure relates to a work analysis apparatus and method.

[0002] Patent Literature 1 discloses a work management system that delivers video information of each process captured by a camera to a line manager at a production site. The work management system measures the work time of each process using measuring means including switches, sensors, etc., and when it determines that the measured work time is shorter or longer than a standard time set internally by a predetermined amount or more, it displays video information of the work corresponding to that work time on a monitor for the manager. In this way, the work management system aims to enable efficient line management by allowing the line manager to view only the video of abnormal conditions, without having to check all the video for abnormal conditions in the process.

[0003] Japanese Patent Application Laid-Open No. 2008-059322

[0004] The present disclosure provides a task analysis device and method that can facilitate the analysis of tasks from video captured by a camera of the workplace where the tasks are performed.

[0005] One aspect of the present disclosure provides a task analysis device for analyzing video captured by one or more cameras of a workplace where multiple tasks are performed. The task analysis device includes a memory unit and a control unit. The memory unit stores video data representing captured video of a work area where each task is performed in the workplace. The control unit detects an abnormal period in which the progress of each task deviates from a predetermined standard for each task based on task information representing the progress of the multiple tasks. Based on the detected abnormal period, the control unit extracts, from the captured video represented by the video data, a first video of a first work area including a location where a first task was performed during the abnormal period, and a second video related to the first video. The second video includes at least one of an image capture result of a second work area located near the first work area and an image capture result of a second task performed at least either before or after the first task.

[0006] These general and specific aspects may be realized by a system, a method, and a computer program, as well as combinations thereof.

[0007] The work analysis device and method disclosed herein can facilitate the analysis of work from video captured by a camera of the workplace where the work is performed.

[0008] FIG. 1 is a diagram showing an overview of a work analysis system according to embodiment 1. FIG. 2 is a block diagram illustrating the configuration of a work analysis device in the work analysis system. FIG. 3 is a functional block diagram illustrating the functional configuration of the work analysis device of embodiment 1. FIG. 4 is a diagram explaining information stored in the memory unit of the work analysis device. FIG. 5 is a diagram explaining an overview of the operation of the work analysis system in embodiment 1. Flowchart illustrating the process of generating an extracted video in the work analysis device of embodiment 1. Diagram for explaining the process of generating an extracted video in embodiment 1. Flowchart illustrating the process of generating an extracted video in variant 1 of embodiment 1. Diagram for explaining the process of generating an extracted video in variant 1. Flowchart illustrating the process of generating an extracted video in variant 2 of embodiment 1. Diagram for explaining an overview of the operation of the work analysis system in embodiment 2. Flowchart illustrating the process of detecting an abnormal period in the work analysis device of embodiment 2. Diagram for explaining the generation of an extracted video in the work analysis device of embodiment 2. Diagram for explaining a workplace to which the work analysis system is applied in a further variant of embodiment 1.

[0009] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventor(s) provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0010] (First embodiment) 1. Configuration An operation analysis system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an overview of an operation analysis system 1 according to this embodiment.

[0011] 1-1. System Overview The work analysis system 1 illustrated in Figure 1 includes multiple cameras 2-1, 2-2, and 2-3, and a work analysis device 5. This system 1 is used to analyze the efficiency of workers W1 and other workers performing work in a workplace 6, such as a factory. This system 1 may also include a monitor 4 for presenting an analysis screen or other image used in the analysis to a user 3, such as a manager or analyst of the workplace 6. In the example of Figure 1, this system 1 further includes multiple sensors 9-1, 9-2, and 9-3.

[0012] In the example of FIG. 1 , a conveyor line 61 is installed in a work area 6, and an item 8, such as a product, is conveyed on the conveyor line 61. In the work area 6 of this example, a worker W1 moves along the conveyor line 61, and sequentially and repeatedly performs a plurality of work processes A, B, and C. For example, in the work of manufacturing the item 8, each of the processes A to C includes picking, assembling, and inspecting parts. In this embodiment, cameras 2-1, 2-2, and 2-3 correspond to processes A, B, and C, respectively, and are positioned so as to capture images of the ranges in the work area 6 where each of the processes A, B, and C is performed. Hereinafter, cameras 2-1 to 2-3 will also be collectively referred to as cameras 2.

[0013] The camera 2 of the system 1 repeatedly captures images at a predetermined interval, for example, in a workplace 6, and generates video data representing a video made up of the captured images. The camera 2 is connected to the work analysis device 5, for example, so that the video data is transmitted to the work analysis device 5.

[0014] Sensors 9-1, 9-2, and 9-3 of this system 1 correspond to processes A, B, and C, respectively, and are positioned at the positions in the workplace 6 where processes A to C end so as to detect items 8 passing on the conveyor line 61. Hereinafter, sensors 9-1 to 9-3 will be collectively referred to as sensors 9. Sensor 9 is, for example, a photoelectric sensor such as an infrared sensor, and is connected to the work analysis device 5 so that sensor data indicating the detection results of the items 8 is transmitted to the work analysis device 5.

[0015] In this system 1, the work analysis device 5 analyzes the work duration, which is the length of time that a worker W1 spent working for each of processes A to C, and various information such as the work duration is presented to the user 3 on the monitor 4. From the presented work duration, the user 3 can analyze abnormal conditions, such as a work duration that is longer than a predetermined standard. Furthermore, in this system 1, the user 3 can analyze the video of each of processes A to C captured by each camera 2 and consider the causes of the abnormal condition.

[0016] 1 illustrates one worker W1, the number of workers in the workplace 6 may be two or more. In this case, the work analysis device 5 may analyze the work period and the like for each worker. The system 1 may also analyze abnormal conditions and the like during the work period for each worker.

[0017] The work analysis device 5 is configured as an information processing device such as a server device. The work analysis device 5 is connected to an external information processing device such as a PC including a monitor 4 so that it can communicate with the external information processing device. The configuration of the work analysis device 5 will be described with reference to FIG. 2.

[0018] 1-2. Configuration of the work analysis device Fig. 2 is a block diagram illustrating the configuration of the work analysis device 5. The work analysis device 5 illustrated in Fig. 2 includes a control unit 50, a storage unit 51, an operation unit 53, a device interface 54, and an output interface 55. Hereinafter, interface will be abbreviated as "I / F."

[0019] The control unit 50 includes, for example, a CPU or MPU that works in cooperation with software to realize predetermined functions, and controls the overall operation of the work analysis device 5. The control unit 50 reads data and programs stored in the memory unit 51, performs various arithmetic processing, and realizes various functions.

[0020] The control unit 50 executes, for example, a program including a set of instructions for realizing the functions of the work analysis device 5. The program may be provided via a communication network such as the Internet, or may be stored on a portable recording medium. The control unit 50 may also have internal memory as a temporary storage area for storing various data and programs.

[0021] The control unit 50 may be a dedicated electronic circuit designed to realize a predetermined function or a hardware circuit such as a reconfigurable electronic circuit. The control unit 50 may be configured with various semiconductor integrated circuits such as a CPU, an MPU, a GPU, a GPGPU, a TPU, a microcomputer, a DSP, an FPGA, and an ASIC.

[0022] The memory unit 51 is a storage medium that stores programs and data necessary to realize the functions of the work analysis device 5. As shown in Fig. 2, the memory unit 51 includes a storage unit 51a and a temporary storage unit 51b.

[0023] The storage unit 51a stores parameters, data, control programs, etc. for implementing predetermined functions. The storage unit 51a is configured, for example, with an HDD or SSD. For example, the storage unit 51a stores the above-mentioned programs, as well as video data D0, work information D1, normal state information D2, and process information D3 (see FIG. 3 ), which will be described later.

[0024] The temporary storage unit 51b is configured with a RAM such as a DRAM or an SRAM, and temporarily stores (i.e., holds) data. The temporary storage unit 51b holds, for example, the detection results of an abnormal period, which will be described later. The temporary storage unit 51b may also function as a work area for the control unit 50, and may be configured as a storage area in the internal memory of the control unit 50.

[0025] The operation unit 53 is a general term for operation members that accept operations by the user 3. The operation unit 53 is configured by, for example, any one of a keyboard, a mouse, a trackpad, a touchpad, buttons, switches, etc., or a combination of these. The operation unit 53 acquires various information input by operations by the user 3.

[0026] The device I / F 54 is a circuit for connecting an external device such as the camera 2 to the work analysis device 5. The device I / F 54 communicates according to a predetermined communication standard. Examples of the predetermined standard include USB, HDMI (registered trademark), IEEE 1395, IEEE 802.11, Bluetooth (registered trademark), etc. The device I / F 54 is an example of an acquisition unit in the work analysis device 5 that receives various information from external devices. In the work analysis system 1, the work analysis device 5 acquires video data representing video captured by the camera 2, for example, via the device I / F 54.

[0027] The output I / F 55 is a circuit for outputting information. The output I / F 55 outputs video data and the like to an external display device such as a monitor or projector for displaying various types of information, in compliance with, for example, the HDMI standard.

[0028] The above-described configuration of the work analysis device 5 is merely an example, and the configuration of the work analysis device 5 is not limited to this. The work analysis device 5 may be configured with various types of computers, including a PC (personal computer). In addition to or instead of the output I / F 55, the work analysis device 5 may also include a display unit configured, for example, as a built-in display device, such as a liquid crystal display or organic EL display. Furthermore, the work analysis method of this embodiment may be executed using distributed computing.

[0029] In addition to or instead of the above configuration, the work analysis device 5 may be configured to communicate with an external information processing device via a communications network. For example, the operation unit 53 may be configured to accept operations from an external information processing device connected via the communications network. The output I / F 55 may also transmit various types of information to the external information processing device via the communications network.

[0030] The acquisition unit in the work analysis device 5 may also be realized in cooperation with various software in the control unit 50, etc. The acquisition unit in the work analysis device 5 may acquire various pieces of information by reading out the information stored in various storage media (e.g., the storage unit 51) into the work area of ​​the control unit 50.

[0031] 1-3. Functional Configuration of the Work Analysis Device The functional configuration of the work analysis device 5 of this embodiment will be described with reference to FIG.

[0032] 3 is a functional block diagram illustrating the functional configuration of the work analysis device 5 of this embodiment. The work analysis device 5 includes a video acquisition unit 501, a task detection unit 502, an anomaly detection unit 503, and a video extraction unit 504, which function in cooperation with software in, for example, the control unit 50. Furthermore, in the work analysis device 5, normal state information D2 and process information D3 are stored in advance, for example, in the storage unit 51a of the memory unit 51. The normal state information D2 indicates predetermined standards that are assumed to indicate that work by worker W1 and others in the workplace 6 has been performed normally. The process information D3 indicates the relationship between the locations in the workplace 6 where processes A to C are performed and each camera 2, etc.

[0033] The video acquisition unit 501 sequentially acquires video data D0 indicating videos captured by each camera 2 at a video frame period or the like, for example, from each camera 2 via the device I / F 54 or the like. The video acquisition unit 501 stores the acquired video data D0 in association with the imaging time of each frame, for example, in the storage unit 51, and inputs the acquired video data D0 to the work detection unit 502.

[0034] The work detection unit 502 detects the work process shown in each frame image, for example, based on the video data D0 input for each frame. For example, the work detection unit 502 uses various image recognition algorithms to detect whether or not each process A to C is shown in the frame image. Based on the detection results obtained sequentially, the work detection unit 502 generates work information D1 that indicates the progress of the work, such as whether or not work is being performed at each time in the workplace 6, and stores the work information D1 in the storage unit 51.

[0035] The abnormality detection unit 503 detects an abnormal period in which an abnormal condition occurred in the work performed by the worker W1, etc., based on, for example, the work information D1 and normal state information D2 acquired from the storage unit 51. The abnormal period is detected as, for example, a period in which the progress of the work performed by the worker W1, etc., deviates from the standard of the normal state information D2. Based on the detected abnormal period and process information D3, the abnormality detection unit 503 outputs abnormality information that associates the abnormal period, the process performed during the abnormal period, and the camera 2 corresponding to the process to the video extraction unit 504.

[0036] The video extraction unit 504 generates an extracted video by extracting videos captured by some of the cameras 2 from the videos captured by each camera 2 during a predetermined analysis period, depending on the detected abnormal period, etc., based on the anomaly information from the anomaly detection unit 503, the video data D0 and process information D3 in the storage unit 51. The analysis period is the period to be analyzed using the cameras 2 in the work analysis system 1, and is set in advance, for example, to one day. The analysis period may be any period specified by input from the user via the operation unit 53, and may be one hour, one week, or one month.

[0037] 1-4. Normal State Information and Process Information Fig. 4 is a diagram for explaining the information stored in the storage unit 51 of the work analysis device 5. Fig. 4(A) illustrates normal state information D2 for processes A to C. Fig. 4(B) illustrates process information D3 for processes A to C.

[0038] The normal state information D2 includes, for example, the items of "task sequence," "standard work period," and "task interval" as shown in Fig. 4A. The normal state information D2 includes, for each item, a reference value set according to, for example, the work plan in the workplace 6 or the work tendencies of the worker W1 or the like. The value of the normal state information D2 may be set for each worker, such as the worker W1.

[0039] The work sequence indicates the execution order of processes A to C in the workplace 6. In the example of FIG. 1, work is performed repeatedly in the order of processes A, B, and C. The standard work period indicates the time required for a worker W1 or the like to perform each of processes A to C, and is calculated, for example, by averaging periods measured in advance multiple times. In the normal state information D2 of FIG. 4A, standard work periods T1, T2, and T3 are set for each of processes A, B, and C, respectively. The work interval indicates the period from the end of the previous process to the start of the next process for each two adjacent processes A to C in the work sequence, and is set to "0" in both cases in FIG. 4A.

[0040] 4(B), the process information D3 includes a "camera number" that identifies each camera 2 and a "camera position" that indicates the location where each camera 2 is installed in the workplace 6. The process information D3 manages each process A to C in association with one of the cameras 2-1 to 2-3 that captures the area in the workplace 6 where the process is performed, i.e., the work area of ​​each process A to C. In the example of FIG. 1, the process information D3 stores the camera numbers that indicate the corresponding cameras 2-1, 2-2, and 2-3 for each process A, B, and C, respectively.

[0041] In this embodiment, for example, each camera 2 has an angle of view according to the imaging range set for each camera in the work area 6, and the imaging range of each camera 2 corresponds to the camera position in the process information D3. Furthermore, the process information D3 in this embodiment manages information related to each process in the order of processes A, B, and C, for example, according to the work sequence in the normal state information D2, and may be used, for example, to determine the work sequence in the processing described below.

[0042] 2. Operation The operation of the work analysis system 1 and work analysis device 5 configured as above will now be described.

[0043] 2-1. Overview of Operation An overview of the operation of the task analysis system 1 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram for explaining the overview of the operation of the task analysis system 1 in this embodiment.

[0044] 5 shows an example in which the work analysis device 5 of the system 1 illustrated in FIG. 1 detects the work of processes A to C performed by a worker W1 in a workplace 6. The work analysis device 5 accumulates video data D0 of the workplace 6 captured by each camera 2 over an analysis period, such as one day, in the storage unit 51a. Based on the results of work detection, for example, by image recognition processing of the accumulated video data D0, the work analysis device 5 outputs a work time graph that visualizes the work period and displays it on the monitor 4. As shown in FIG. 5, the work time graph chronologically shows the work period of the worker W1 for each process A to C.

[0045] The system 1 further detects abnormal periods in the work of worker W1 and displays each of the abnormal periods A1 to A4 on a work time graph, as shown in Fig. 5, for example. In the example of Fig. 5, in the work of worker W1 detected for each of processes A to C, abnormal period A1 corresponds to an abnormal state in which the work period is longer than the standard work period for process B, and abnormal period A2 corresponds to an abnormal state in which the work period is shorter than the standard work period for process C. Furthermore, abnormal period A3 corresponds to an abnormal state in which no work is performed between process C and the following process A, and abnormal period A4 corresponds to an abnormal state in which process B following process A is not performed, resulting in an incomplete work.

[0046] The system 1 displays, for example, video from cameras 2 corresponding to abnormal periods A1 to A4 in a video display area 73 on a monitor 4. At this time, the work analysis device 5 extracts video from some of the cameras 2 corresponding to each of the abnormal periods A1 to A4 from among the multiple cameras 2. With this display, the user 3 can check the video displayed in the video display area 73 for each of the abnormal periods A1 to A4, for example, and analyze details such as the cause of the abnormal state in the work of worker W1. Furthermore, rather than displaying all of the video from each camera 2, the work analysis device 5 can simply display some of the video for each of the abnormal periods A1 to A4, thereby reducing the processing load.

[0047] Here, for example, it is conceivable that the cause of the abnormal state during the abnormal period lies in the work order in the workplace 6 or in the surrounding processes in terms of the positional relationship in which each process is performed.

[0048] Therefore, the work analysis system 1 of this embodiment extracts, for example, from the video footage captured by each camera 2, video footage from the cameras 2 corresponding to processes performed during periods before, after, etc., of the abnormal periods A1 to A4 among processes A to C. With this system 1, for example, the user 3 can analyze each of the abnormal periods A1 to A4 by checking the extracted video footage, making it easier to analyze the work performed in the workplace 6 from the video footage captured by each camera 2 of the workplace 6.

[0049] 2-2. Operation of the work analysis device The operation of the work analysis device 5 in the work analysis system 1 will be described below.

[0050] 2-2-1. Video Acquisition and Task Detection The task detection operation in which the task analysis device 5 acquires video data D0 of the workplace 6 from the camera 2 and detects tasks of each of processes A to C performed in the workplace 6 through image recognition processing will be described with reference to FIG. 6 .

[0051] 6 is a flowchart illustrating the operation of the video acquisition unit 501 and the task detection unit 502 in the work analysis device 5 of this embodiment. The processing of this flowchart is executed by the control unit 50 of the work analysis device 5 functioning as the video acquisition unit 501 and the task detection unit 502 shown in FIG. 3. The processing of this flowchart is executed repeatedly, for example, every analysis period.

[0052] First, the control unit 50, as the video acquisition unit 501, acquires image data for each frame of a video captured by one camera 2 via the device I / F 54 (S1). For example, in the work analysis system 1 of Fig. 1, image data is first acquired from the camera 2-1.

[0053] Next, the control unit 50, as the task detection unit 502, detects the task steps performed by each worker, such as worker W1, who appears in the frame image, based on the acquired image data, through image recognition processing of the frame image (S2). For example, the control unit 50 detects task A in the image data from camera 2-1. The control unit 50 may switch between tasks A to C to be detected depending on which of cameras 2-1 to 2-3 the image data was acquired from.

[0054] The control unit 50 associates the image capturing time of the frame image with the detected work step and records the associated image in work information D1 for each worker (S3). The work information D1 is stored in, for example, the temporary storage unit 51b. For example, if the target work step is not detected in step S2, the image capturing time may be associated with information indicating non-detection and recorded in the work information D1.

[0055] The control unit 50, for example, refers to the process information D3 stored in the storage unit 51a and determines whether image data of a frame captured at the same time as the frame captured in step S1 has been acquired for all cameras 2 in the workplace 6 (S4).

[0056] If image data at the same image capture time has not been acquired for all cameras 2 (NO in S4), the control unit 50 repeats the processes of steps S1 to S3. In the example of Fig. 1, steps S1 to S3 are repeated until image data at the same image capture time has been acquired for all cameras 2-1 to 2-3 (NO in S4).

[0057] When image data at the same image capture time has been acquired for all cameras 2 (YES in S4), the control unit 50 determines whether image data for each frame in the analysis period has been acquired, i.e., whether acquisition of the video captured by each camera 2 in the analysis period has been completed (S5). The control unit 50 makes the determination in step S5 based on, for example, the length of the analysis period set in advance and the frame rate of the video.

[0058] If the acquisition of the video for the analysis period has not been completed (NO in S5), the control unit 50 repeats the processes from step S1 onwards.

[0059] When the acquisition of the video for the analysis period is completed (YES in S5), the control unit 50 stores the work information D1 recorded in step S3 and the video for each camera 2 in which image data for each frame is acquired for each camera 2 in the storage unit 51a (S6). Thereafter, the control unit 50 ends the processing of this flowchart.

[0060] According to the above process, video from each camera 2 during the analysis period is acquired (S1, S4, S5), and based on the results of detecting work processes A to C in each frame image of the video, work information D1 for each time point and the video for each camera 2 are saved (S2, S3, S6). This allows the video to be used for analysis in the work analysis system 1 to be acquired, and also provides chronological work information D1 in which processes A to C are detected for each frame of the video.

[0061] In the above, an example has been described in which the work step performed by worker W1 is detected by image recognition processing. When multiple workers are performing work, each worker may be detected separately by image recognition processing, or each worker may be identified according to their position in the workplace 6. Furthermore, each worker may be identified based on personal authentication information, for example, when the worker enters the workplace 6.

[0062] Furthermore, the operation of task detection is not limited to the above example, and may involve, for example, acquiring video data for an analysis period from camera 2-1, generating task information D1 by performing image recognition processing for each frame image of camera 2-1, and then performing similar processing for cameras 2-2 and 2-3. In step S1, video acquisition unit 501 may acquire video data that has been received in advance from camera 2 and stored in storage unit 51a, for example.

[0063] 2-2-2. Detection of Abnormal Periods and Generation of Extracted Video After the above-described task detection operations, the task analysis device 5 detects abnormal periods in the task and generates extracted videos from the video data D0 in accordance with the detected abnormal periods. This will be described with reference to FIGS. 3 and 7 to 9.

[0064] 7 is a flowchart illustrating the operation of the anomaly detection unit 503 and video extraction unit 504 in the work analysis device 5 of this embodiment. The processing of this flowchart is executed by the control unit 50 of the work analysis device 5 functioning as the anomaly detection unit 503 and video extraction unit 504 shown in FIG. 3. The processing of this flowchart starts when, for example, the storage unit 51a has stored therein the video data D0 and work information D1 obtained by the processing shown in the flowchart of FIG. 6 in addition to the normal state information D2 and process information D3.

[0065] First, the control unit 50 acquires from the storage unit 51a the video data D0 and work information D1 saved in step S6 of FIG. 6, as well as the stored normal state information D2 and process information D3 (S11).

[0066] Next, the control unit 50, functioning as the abnormality detection unit 503, detects an abnormal period in the work of the worker W1, etc., based on the work information D1, the normal state information D2, and the process information D3 (S12). By this abnormal period detection process (S12), for example, abnormal periods A1 to A4, etc., shown in FIG. 5 are detected. The control unit 50 stores abnormal information including the detected abnormal periods A1 to A4, etc., in the temporary storage unit 51b. The abnormal period detection process (S12) will be described in detail later.

[0067] Based on the work information D1 and the abnormality information obtained in step S12, the control unit 50 causes the monitor 4 to display a setting screen for user settings related to generation of the extracted moving image (S13), for example, via the output I / F 55. An example of the display of such a setting screen is shown in FIG.

[0068] FIG. 8 is a diagram illustrating a setting screen in the work analysis system 1. FIG. 8A illustrates a setting screen 71 displayed on the monitor 4 in step S13 of FIG. 7. The setting screen 71 displays, for example, a work time graph of worker W1 similar to that shown in FIG. 5, as well as setting items for setting extraction conditions for generating an extracted video. The extraction conditions include, for example, a threshold for determining whether or not to detect an abnormal work period based on the difference (also referred to as "deviation") between the work period of worker W1 for each of processes A to C or the interval between each of processes A to C and a predetermined reference value. In FIG. 8A, the threshold is set to "3" minutes.

[0069] 8A , the extraction conditions include whether to generate an extracted video including periods before and after the abnormal period, the length of the period if the periods before and after are included, and whether the abnormal period includes a missed operation in which a process included in the operation is not performed. In the example of FIG. 8A , the setting screen 71 indicates that, in the currently displayed settings, a total of 20 minutes of video is extracted from the video data D0, and that there are three scenes corresponding to each abnormal period. The extraction conditions may be set to, for example, initial values ​​stored in the storage unit 51 a.

[0070] 8A is displayed, the control unit 50 determines whether a user operation to set video extraction conditions has been input through the operation unit 53 (S14). While the user operation is not input (NO in S14), the control unit 50 repeats the determination in step S14. Note that if the user operation is not input for a predetermined period of time, the control unit 50 may proceed to step S15.

[0071] 8(B) shows an example in which a user operation to set video extraction conditions is input (YES in S14) and the extraction conditions are changed from the example in FIG. 8(A). In the example in FIG. 8(B), the deviation from the reference value is changed from "3 minutes" in the example in FIG. 8(A) to "5 minutes." In accordance with this change, on the setting screen 71 in FIG. 8(B), the total duration of videos extracted from video data D0 is "10 minutes," and the number of scenes corresponding to each abnormal period is "2," both of which are decreased from the example in FIG. 8(A).

[0072] If the user operation is input (YES in S14), the control unit 50, for example as the video extraction unit 504, generates an extracted video for one abnormal period in the abnormality information based on the video data D0, process information D3, and abnormality information (S15). In this extracted video generation process (S15), the control unit 50 generates an extracted video for each abnormal period in accordance with the extraction conditions set by the user operation in step S14, for example. The extracted video generation process (S15) will be described in detail later.

[0073] The control unit 50 determines whether or not there is any abnormal period for which the extracted moving image generation process (S15) has not been applied, i.e., for which no moving image has yet been extracted, among the abnormal periods detected in step S12 and included in the abnormality information (S16). The control unit 50 may manage, for example, in the temporary storage unit 51b, a flag or the like indicating whether or not a moving image has been extracted for each of the abnormal periods A1 to A4, by associating the abnormal periods with each other.

[0074] If there is an abnormal period for which no moving image has been extracted yet (YES in S16), the control unit 50 executes the process of generating an extracted moving image (S15) for that abnormal period.

[0075] If there is no abnormal period during which no video has been extracted (NO in S16), the control unit 50 displays an analysis screen on the monitor 4 via the output I / F 55 based on the work information D1, the abnormality information, and the video data of the generated extracted video (S17). Figure 9 is a diagram for explaining the analysis screen 7 in the work analysis system 1.

[0076] The analysis screen 7 displays, in addition to the same display content as the setting screen 71 of FIG. 8A , a video display area 73 and a link for displaying the extracted video, etc. on the monitor 4. The video display area 73 displays a list of scenes corresponding to the extracted video for each abnormal period using thumbnail images, etc. In this list display, the corresponding extracted video may be played on the monitor 4 in response to, for example, a click on a thumbnail image using the operation unit 53, such as a mouse. The analysis screen 7 also displays a link for displaying a video combining the extracted videos of each scene. The control unit 50 may generate such a scene-combined video by combining the extracted videos in chronological order, for example, after the extracted video generation process (S15).

[0077] According to the analysis screen 7, a user 3 of the present system 1 can analyze extracted videos corresponding to each abnormal state or analyze scene-combined videos, for example, to examine detected abnormal periods, etc.

[0078] After displaying the analysis screen 7 (S17), the control unit 50 ends the processing of this flowchart.

[0079] According to the above process, abnormal periods in work are detected based on the work information D1, etc. (S12), and from the videos captured by the multiple cameras 2, videos from the cameras 2 corresponding to the process performed during each abnormal period and the processes before and after each abnormal period are extracted (S15). This allows the user 3, for example, to simply check the extracted videos for the abnormal conditions that occurred during each abnormal period, making it easier to analyze the work being performed in the workplace 6 from the videos captured by the cameras 2 of the workplace 6.

[0080] In the above process, in response to a user operation for setting extraction conditions for a video (S14), an extracted video is generated according to the extraction conditions set by the user (S15). This makes it easier for the user 3 to analyze the work from the extracted video generated according to the desired extraction conditions.

[0081] Furthermore, for example, the analysis screen 7 is displayed on the monitor 4 or the like so as to present the generated extracted videos, etc. to the user 3 (S17). This allows the user 3 to check each extracted video, etc. on the analysis screen 7, for example.

[0082] 2-2-2-1. Abnormal Period Detection Processing The abnormal period detection processing in step S12 in FIG. 7 will be described in detail with reference to FIG.

[0083] 10 is a flowchart illustrating the abnormal period detection process (S12) in the work analysis device 5 of this embodiment. The process of this flowchart starts with the work information D1, normal state information D2, and process information D3 acquired in step S11 of FIG. 7 being stored in the temporary storage unit 51b or the like. Each process of this flowchart is executed by the control unit 50 functioning as the abnormality detection unit 503 shown in FIG. 3, for example.

[0084] The control unit 50 acquires work information of one worker during an analysis period from work information D1 recorded for each worker, for example, and stores it in an internal memory (S21). For example, work information about worker W1 is acquired first.

[0085] The control unit 50 selects one process from the processes recorded in chronological order in the acquired work information (S22). For example, first, the process corresponding to the earliest time in the work information is selected.

[0086] Based on the normal state information D2, the control unit 50 determines whether the process selected in the work information is being performed in accordance with a predetermined work order starting from the previous process that was performed immediately before the selected process (S23). Specifically, the control unit 50 determines whether the order of the selected process and the previous process matches the work order in the normal state information D2 (S23). In the example of the abnormal period A4 in Figure 5, when process C is selected in step 22, the control unit 50 determines that process C is in an abnormal state, that is, it does not follow the normal order starting from the previous process A in the work information (NO in S23).

[0087] If the selected process does not follow the predetermined work order (NO in S23), the control unit 50 stores in the temporary storage unit 51b abnormal information that associates the abnormal period in which the abnormal condition occurred, the process performed during the abnormal period, and the camera number of the camera 2 corresponding to that process (S26). The abnormal period is determined, for example, according to the abnormal condition, and in the case of an abnormal condition in which the process order in the work information differs from the work order in the normal state information D2, i.e., in which a process is omitted (NO in S23), the abnormal period may be determined to be the period of the process in which a process is omitted based on the work information.

[0088] For example, in the abnormal period A4 in FIG. 5, the period of process B, which is performed between processes A and C in the work sequence of the normal state information D2, is determined to be the abnormal period, and the start and end times of that period are recorded in the abnormal information. In this case, the control unit 50, for example, references the normal state information D2 to identify the process performed during the abnormal period (also referred to as the "abnormal process") as process B. The control unit 50 also identifies the camera number of camera 2 corresponding to the abnormal process based on the process information D3. For example, in the process information D3 in FIG. 4(B), camera 2-2 with camera number "2" corresponds to abnormal process B.

[0089] After storing the abnormality information in the temporary storage unit 51b (S26), the control unit 50 determines whether or not there is a process next to the selected process in the work information acquired in step S21 (S27). If there is a next process in the work information (YES in S27), the control unit 50 returns to step S22, selects the next process, and repeats the processing from step S23 onwards.

[0090] If the selected process is being performed in the normal work order from the previous process (YES in S23), the control unit 50 determines whether the difference between the duration of the selected process in the work information and the standard work period of the process in the normal state information D2 is equal to or greater than a predetermined value (S24). The predetermined value is set to, for example, 1 minute, from the perspective of detecting abnormal periods, such as abnormal periods A1 and A2 in FIG. 5, in which the work period is too long or too short compared to the standard work period. The predetermined value may vary depending on the type of previous process and the selected process, such as 30 seconds or 10 minutes. The predetermined value may also vary depending on whether an abnormal period in which the work period is longer or shorter than the standard work period is to be detected. The predetermined value may be a value different from the user setting in step S14 in FIG. 7.

[0091] If the difference between the period of the selected process and the standard work period is equal to or greater than a predetermined value (YES in S24), the control unit 50 stores the abnormal period, etc. as abnormality information in the temporary storage unit 51b (S26). In the example of the abnormal period A1, the period of process B in the work information is stored as the abnormal period, and process B is stored as the abnormal process.

[0092] If the difference between the duration of the selected process and the standard work duration is not equal to or greater than the predetermined value (NO in S24), the control unit 50 determines whether the interval between the start time of the selected process in the work information and the end time of the previous process is equal to or greater than the predetermined interval (S25). The predetermined interval is, for example, the work interval in the normal state information D2. For example, if no work is performed between consecutive processes, as in the abnormal period A3 in Figure 5, it is determined that the interval between the processes is equal to or greater than the predetermined interval (YES in S25).

[0093] If the interval between the start time of the selected process and the end time of the previous process is equal to or greater than the predetermined interval (YES in S25), the control unit 50 proceeds to step S26 and stores the abnormality information in the temporary storage unit 51b. In the example of abnormal period A3 in Fig. 5, when process A is selected, if the interval between the start time of process A and the end time of the previous process C is equal to or greater than the predetermined interval (YES in S25), the abnormality information is stored with the interval as the abnormal period. In this case, the abnormal process may be determined to be, for example, the selected process A or the previous process C.

[0094] If the interval between the start time of the selected process and the end time of the previous process is not equal to or greater than the predetermined interval (NO in S25), the control unit 50 determines whether there is a next process in the work information (S27), as in the case after the abnormal state is maintained (S26).

[0095] If the last process in the work information is selected and there is no next process (NO in S27), the control unit 50 determines whether work information D1 for all workers contains work information for a worker other than the worker whose work information was obtained in step S21 (S28).

[0096] If the work information D1 includes work information of another worker (YES in S28), the control unit 50 returns to step S21, acquires work information of one of the other workers, and repeats the processing from step S22 onwards.

[0097] On the other hand, if the work information D1 does not include work information of another worker (NO in S28), the control unit 50 as the abnormality detection unit 503 outputs the abnormality information stored in the temporary memory unit 51b, for example, to the video extraction unit 504 (S29).

[0098] After outputting the abnormality information (S29), the control unit 50 ends the processing of this flowchart.

[0099] According to the above process, based on the work information D1 and the normal state information D2, it is determined whether the processes detected as work by each worker conform to a predetermined work sequence (S23). It is also determined whether the difference between the duration of each detected process and the standard work period is equal to or greater than a predetermined value (S24), and whether the interval between processes is equal to or greater than a predetermined interval (S25). If the detected process does not conform to the predetermined work sequence (NO in S23), if the difference between the duration of each process and the standard work period is equal to or greater than a predetermined value (YES in S24), or if the interval between processes is equal to or greater than a predetermined interval (YES in S25), abnormality information is stored (S26). The abnormality information includes the abnormal period, the abnormal process, and the camera number corresponding to the abnormal process, all of which are associated with each other.

[0100] According to the above processing, abnormal periods A1 to A4, etc., as shown in FIG. 5 can be detected, and the abnormality information can be used to associate and manage the abnormal processes carried out during each abnormal period with the cameras 2 corresponding to each abnormal process.

[0101] 2-2-2-2 Extracted Moving Image Generation Process The extracted moving image generation process in step S15 of FIG. 7, which uses the abnormality information obtained by the abnormal period detection process (S12) described above, will be described in detail with reference to FIGS. 11 and 12.

[0102] Fig. 11 is a flowchart illustrating the extracted video generation process (S15) in the work analysis device 5 of this embodiment. The process of this flowchart is executed for each detected abnormal period by the control unit 50 functioning as the video extraction unit 504 of Fig. 3, for example, based on the abnormality information output from the abnormality detection unit 503. The process of this flowchart is executed by referencing, for example, the video data D0 and process information D3 acquired in step S11 of Fig. 7, and the abnormality information held in step S12, from the storage unit 51. Fig. 12 is a diagram for explaining the extracted video generation process (S15) in this embodiment.

[0103] First, the control unit 50 identifies the preceding process that was performed immediately before the abnormal process in the workplace 6 and the subsequent process that was performed immediately after the abnormal process, for example, based on the abnormality information and the process information D3 (S41). For example, in the abnormal period A1 in Fig. 5, since the abnormal process is process B, process A is identified as the preceding process and process C is identified as the subsequent process based on the order defined between processes A to C. Furthermore, in the case of a task in which processes A to C are repeated, if process C is the abnormal process, process A is identified as the subsequent process and process B is identified as the preceding process.

[0104] Next, the control unit 50 identifies the cameras 2 corresponding to the upstream process and the downstream process based on the process information D3 (S42). In the example of the abnormal period A1, the process information D3 in FIG. 4B identifies the camera 2-1 corresponding to the upstream process A and the camera 2-3 corresponding to the downstream process C.

[0105] Based on the abnormality information and video data D0, the control unit 50 extracts a video of an abnormal period from the videos captured by the cameras 2 corresponding to the abnormal process, for example, from the videos captured by each camera 2 in the workplace 6 of Figure 1 (S43). Figure 12 shows an example of an extracted video V6 generated for the abnormal period A1. In the example of Figure 12, a video V2 of the abnormal period A1 is extracted from the videos captured by the camera 2-2 corresponding to the abnormal process B.

[0106] Based on the video data D0, the control unit 50 extracts a video of a predetermined period of N seconds (N is a positive number) immediately before the abnormal process from the video from each camera 2 corresponding to the previous process (S44). The predetermined period is set, for example, on the setting screen 71 in Fig. 8 in response to the user operation in step S14 in Fig. 7. In the example in Fig. 12, a video V1 of a predetermined period is extracted from the video from the camera 2-1 corresponding to the previous process A.

[0107] Based on the video data D0, the control unit 50 extracts a video for a predetermined period of N seconds immediately after the abnormal process from the video from each camera 2 corresponding to the subsequent process (S45). In the example of Fig. 12, a video V3 for a predetermined period is extracted from the video from camera 2-3 corresponding to subsequent process B.

[0108] The control unit 50 generates an extracted video by combining the extracted videos of the preceding process, the abnormal period, and the following process (S46). For example, the control unit 50 combines the videos in chronological order. In the example of FIG. 12, the video V1 of a predetermined period in the preceding process, the video V2 of the abnormal period A1, and the video V3 of a predetermined period in the following process are combined in that order to generate an extracted video V6. Furthermore, instead of generating the extracted video V6, the control unit 50 may also display the videos V1 to V3 captured by the cameras 2 corresponding to the preceding process, the abnormal process, and the following process on a monitor 4 or the like and play them back in chronological order.

[0109] After generating the extracted moving image (S46), the control unit 50 stores the generated extracted moving image in the storage unit 51a, for example, and ends the processing of this flowchart.

[0110] According to the above process, the cameras 2 corresponding to the processes preceding and succeeding the abnormal process are identified (S41, S42), and videos from the cameras 2 corresponding to the abnormal period, the preceding process, and the succeeding process are extracted from the videos taken by each camera 2 (S43-S45). The extracted videos are then combined to generate an extracted video (S46). In this way, videos from the cameras 2 corresponding to the abnormal process as well as the preceding and succeeding processes can be extracted from the videos taken by each camera 2 in the workplace 6, and an extracted video including the period before and after the abnormal period can be generated.

[0111] In the above process, an example has been described in which videos from cameras 2 corresponding to both the upstream process and downstream process of the abnormal process are used to generate the extracted video (S41, S42, S44, S45). The generation of the extracted video is not limited to the above example, and videos from cameras 2 corresponding to either the upstream process or the downstream process may be used. Furthermore, in steps S44 and S45, videos of different periods may be extracted for the upstream process and the downstream process.

[0112] (Variation 1) In the first embodiment, an example has been described in which videos of processes performed before and after the abnormal period are included in the extracted video in the extracted video generation process (S15). The extracted video may include, for example, videos of processes performed near the position where the abnormal process is performed in the workplace 6. Such variation 1 of the first embodiment will be described using FIGS. 13 and 14 .

[0113] Fig. 13 is a flowchart illustrating the extracted moving image generation process in Modification 1 of Embodiment 1. Fig. 14 is a diagram for explaining the extracted moving image generation process in Modification 1. The process of the flowchart in Fig. 13 is executed, for example, instead of the extracted moving image generation process in Embodiment 1 (S15 in Fig. 7, Fig. 11).

[0114] First, the control unit 50 identifies, among the cameras 2, a nearby camera located near the camera 2 corresponding to the abnormal process in the workplace 6 based on the process information D3 (S51). In this modification, the nearby camera is identified as, for example, a camera 2 whose camera position in the process information D3 is within a predetermined range from the camera position of the camera 2 corresponding to the abnormal process. The predetermined range is set, for example, in the workplace 6, so that the imaging ranges captured by the two cameras 2 are adjacent to each other. The nearby camera may be a camera corresponding to a process before or after the abnormal process in the workplace 6, or a camera corresponding to the process two processes before or two processes after the abnormal process, or a camera corresponding to the process three processes before or three processes after the abnormal process.

[0115] 14 illustrates the imaging ranges 21, 22, and 23 of the workplace 6 for the cameras 2-1, 2-2, and 2-3, each of which has the same angle of view. In this example, the positional relationship between the camera positions (x, y), (u, v), and (z, w) of the cameras 2-1, 2-2, and 2-3 corresponds to the positional relationship between the imaging ranges 21, 22, and 23. For example, if the abnormal process is process B, the camera 2-2 corresponds to the abnormal process, and therefore in step S51, the nearby cameras are identified as cameras 2-1 and 2-3, which have imaging ranges 22 and 23 adjacent to the imaging range 22 of camera 2-2.

[0116] Next, the control unit 50 extracts a video taken during the abnormal period from, for example, videos taken by a nearby camera (S52).

[0117] The control unit 50 extracts the video during the abnormal period from the videos captured by the camera 2 corresponding to the abnormal process (S53).

[0118] The control unit 50, for example, refers to the process information D3 and combines the video of the abnormal process extracted in step S53 with the video from the nearby camera extracted in step S52 based on the positions of the camera 2 corresponding to the abnormal process and the nearby camera, to generate an extracted video (S54). For example, the videos are combined so that they are displayed in an arrangement corresponding to the positions of the cameras in the extracted video.

[0119] According to the above process, a camera near the camera 2 corresponding to the abnormal process is identified (S51), video from the camera 2 during the abnormal period and video from the nearby cameras are extracted (S52, S53), and an extracted video is generated by combining the respective videos (S54). Using such extracted video including video from the nearby cameras makes it easier for the user 3 to analyze the work, for example, even when the cause of the abnormal state is found in a process surrounding the location where the abnormal process is performed in the workplace 6.

[0120] (Variation 2) In the above variation 1, an example has been described in which, based on the camera positions in the process information D3, videos from a camera 2 corresponding to the abnormal process and a camera 2 whose imaging range in the workplace 6 is adjacent to each other are included in the extracted videos. The extracted videos may include videos of processes performed at positions where the worker W1 performed work before and after the abnormal process, for example, according to the flow line indicating the trajectory of the worker W1's movement in the workplace 6. Such variation 2 of embodiment 1 will be described using FIG. 15 .

[0121] Fig. 15 is a flowchart illustrating the extracted moving image generation process in Modification 2 of Embodiment 1. Fig. 16 is a flowchart for explaining the extracted moving image generation process in Modification 2. The process of the flowchart in Fig. 15 is executed, for example, instead of the extracted moving image generation process in Embodiment 1 (S15 in Fig. 7, Fig. 11).

[0122] First, the control unit 50 identifies the worker who performed the abnormal process, for example, based on the work information D1 in which the process performed by each worker at each time is recorded (S61).

[0123] Next, the control unit 50 identifies, from among the cameras 2, the camera 2 that captured footage of the worker during a predetermined period before and after the abnormal process, for example, based on the flow line information indicating the worker's flow line identified in step S61 (S62). In this modified example, for example, an omnidirectional camera or the like is installed in the workplace 6 so as to capture images of the entire area in which each worker performs their work. For example, before starting this flowchart, the work analysis device 5 generates flow line information indicating the location of each identified worker at each time point, based on video data indicating videos captured by the omnidirectional camera during the analysis period, by image recognition processing of the videos. The flow line information is stored, for example, in the storage unit 51a.

[0124] In this modification, the imaging range of each camera 2 in the workplace 6 may be stored in the process information D3, etc. In step S62, the control unit 50 identifies the camera 2 having an imaging range that includes the position of the worker, for example, based on the position of the worker in the flow line information.

[0125] The control unit 50 extracts videos from the cameras 2 identified in step S62, each of which is taken during a predetermined period (S63).

[0126] The control unit 50 extracts the video of the abnormal period from the video captured by the camera 2 corresponding to the abnormal process (S64).

[0127] The control unit 50 generates an extracted moving image by combining the moving images extracted in steps S63 and S64, for example, in the order of the time at which the worker identified in step S61 appears (S65).

[0128] According to the above process, an extracted video is generated that includes video of the worker before and after the abnormal process, based on the movement line of the worker who was performing the abnormal process (S61 to S65). Such extracted video makes it easier for user 3 to analyze the work, even when the abnormal state of the abnormal process is caused by the behavior of the worker. In the above process, an example of extracting video before and after the abnormal process has been described (S62, S63), but it is not limited to before and after, and video of either before or after the abnormal process may be extracted.

[0129] In the above explanation of Modifications 1 and 2, examples have been described in which the generation of the extracted video shown in the flowcharts of Figures 13 and 15 is executed instead of the extracted video generation process of embodiment 1 (S15 in Figure 7, Figure 11), but it may also be executed in combination with the process of embodiment 1. Thus, in this embodiment, in addition to the video of the abnormal period, the control unit 50 extracts video from the camera 2 associated with at least one of the processes performed before and after the abnormal process and the processes performed in a work area near the work area where the abnormal process was performed (S44, S45, S52, S63).

[0130] 3. Effects, etc. As described above, the work analysis device 5 of this embodiment is an example of a work analysis device for analyzing video captured by a camera 2 (an example of one or more cameras) of a workplace 6 where processes A to C are performed as an example of multiple tasks. The work analysis device 5 includes a memory unit 51 and a control unit 50. The memory unit 51 stores video data D0 representing captured video of the work area in the workplace 6 where each of processes A to C is performed. Based on work information D1 indicating the progress of the multiple processes A to C, the control unit 50 detects abnormal periods A1 to A4, etc., in which the progress deviates from predetermined standards for each of the multiple processes A to C (S12). Based on the detected abnormal periods, the control unit 50 extracts, from the captured video represented by the video data D0, a video (an example of a first video) of a first work area including a location where the abnormal process (an example of a first task) was performed during the abnormal period, and a second video related to the first video (S15). The second video includes at least one of the imaging results of a second work area located near the first work area and the imaging results of a process (an example of the second work) performed at least either before or after the abnormal process (see Figures 11 to 15, S41 to S42, S44 to S45, S51 to S52, S62 to S63).

[0131] According to the above-described work analysis device 5, an abnormal period in the progress of work performed by worker W1 in each of processes A to C is detected based on work information D1 (S12). Then, for the detected abnormal period, a video of the work area in which the abnormal process was performed during the abnormal period is extracted as a first video from the captured video of the workplace 6 taken by camera 2 (S15). Furthermore, video of a work area located near the work area of ​​the abnormal process and / or a process performed before and / or after the abnormal period is extracted from the captured video as a second video (S15). The extracted first and second videos can facilitate analysis of work performed in workplace 6, even when, for example, factors that cause an abnormal condition exist near the work area of ​​the abnormal process and / or before and / or after the abnormal period.

[0132] In this embodiment, the workplace 6 is imaged by a plurality of cameras 2. The storage unit 51 stores process information D3 as an example of work-related information that associates at least one of the plurality of cameras 2 with each of the plurality of processes A to C according to the work area of ​​each process. Based on the process information D3, the control unit 50 extracts video of the abnormal period from the video captured by a camera 2 of the plurality of cameras 2 associated with the abnormal process (S43, S53, S64), and extracts video of the process from the video captured by a camera 2 associated with at least one of a process performed in a work area (an example of a second work area) near the work area where the abnormal process was performed and a process performed before and after the abnormal period (an example of a second task) (S44, S45, S52, S63) (S15).

[0133] As a result, once a process is identified in the work analysis device 5, videos capturing the work area of ​​that process can be extracted by a simple process that references the process information D3, for example. Furthermore, the user 3 can analyze the extracted videos captured by some of the cameras 2 associated with an abnormal process, etc., out of the multiple cameras 2, making it easier for them to analyze the work.

[0134] In this embodiment, the control unit 50 generates an extracted video V6 by combining the videos V1 to V3 based on video data showing the extracted video V2 (an example of a first video) during the abnormal period and video data showing videos V1 and V3 as examples of second videos related to the first video (S46, see FIG. 12). An extracted video is also generated similarly by the processing of step S54 in FIG. 13 and step S65 in FIG. 15. Using such an extracted video, for example, the user 3 can analyze the video of the abnormal period and videos related to the abnormal period together.

[0135] In the first modification of this embodiment, the video from the nearby camera (an example of the second video) includes a video of a work area adjacent to the work area of ​​the abnormal process (an example of the first work area) in the workplace 6, captured as the second work area (S51, S52). This makes it possible to extract a video of a work area located near the work area of ​​the abnormal process in the workplace 6.

[0136] In the second modification of this embodiment, the control unit 50 extracts videos (an example of second videos) from the camera 2 capturing images of the worker W1 during a predetermined period before and after the abnormal process, based on the flow line information indicating the position of the worker W1 at each time in the workplace 6 (S62, S63). The videos include videos capturing the positions where the worker W1 performed work at least either before or after the abnormal process. This allows for the extraction of videos of processes performed before and / or after the abnormal process at positions on the flow line of the worker W1.

[0137] In this embodiment, the control unit 50 generates task information D1 through image recognition processing based on the video data D0 (S2, S3). The task information D1 includes information indicating which of processes A to C was performed at each time in the workplace 6. This allows the video data D0 to be used not only to present the video data D0 to the user 3 but also to generate the task information D1. In this embodiment, task analysis can be performed even if the task analysis system 1 does not include a separate sensor 9, for example.

[0138] In this embodiment, the work analysis device 5 further includes an operation unit 53 that inputs user operations to set extraction conditions for extracting videos from the captured videos. The control unit 50 extracts videos from the captured videos that are from the abnormal period and nearby videos (examples of the first video and the second video) in response to the user operations input via the operation unit 53 (S14, S15, see FIG. 8 ). This allows the user 3 to perform analysis using the videos extracted from the captured videos in accordance with the desired conditions, making it easier to analyze tasks from the captured videos.

[0139] In this embodiment, the storage unit 51 stores normal state information D2 (an example of work standard information) that includes at least one of the following predetermined standards: a standard work period (an example of the length of time it takes for each of processes A to C to be performed), a work order (an example of the order in which each of processes A to C is performed), and a work interval (an example of the time interval between two processes A to C). The control unit 50 detects an abnormal period by comparing the work information D1 with the normal state information D2 (S12, S23 to S25). The standards in the normal state information D2 may be updated, for example, in response to a predetermined user operation or statistical analysis of the work information D1.

[0140] In this embodiment, the operation of the work analysis device 5 as described above provides a task analysis method for analyzing video captured by a camera 2 (an example of one or more cameras) of a workplace 6 where processes A to C are performed as an example of multiple tasks. The storage unit 51 of the work analysis device 5 (an example of a computer) stores video data D0 representing captured video of the work area in the workplace 6 where each of processes A to C is performed. Based on task information D1 indicating the progress of the multiple processes A to C, the control unit 50 detects abnormal periods A1 to A4, etc., in which the progress deviates from a predetermined standard for each of the multiple processes A to C (S12). Based on the detected abnormal periods, the control unit 50 extracts, from the captured video represented by the video data D0, a video (an example of a first video) of a first task area including a location where the abnormal process (an example of a first task) was performed during the abnormal period, and a second video related to the first video (S15). The second video includes at least one of the imaging results of a second work area located near the first work area and the imaging results of a second work performed at least either before or after the abnormal process (see Figures 11 to 15, S41 to S42, S44 to S45, S51 to S52, S62 to S63).

[0141] In this embodiment, a program is provided for causing the work analysis device 5 to execute the above-described work analysis method as an example of a computer. The above-described work analysis method and program make it easier to analyze work performed in the workplace 6 from video captured by the camera 2.

[0142] (Embodiment 2) In embodiment 1, we have described an work analysis device 5 that detects abnormal periods using work information D1 generated based on video data D0. In embodiment 2, we will describe an work analysis device 5 that detects abnormal periods using work information D1 generated based on sensor data from a sensor 9 in a work analysis system 1.

[0143] Below, the description of the configuration and operation similar to those of the work analysis system 1 and work analysis device 5 according to the first embodiment will be omitted as appropriate, and the work analysis system 1 and work analysis device 5 according to this embodiment will be described.

[0144] 1, sensors 9-1 to 9-3 are provided corresponding to each of processes A to C. In the work analysis device 5 of this embodiment, for example, in process information D3 (see FIG. 4(B)), sensors 9-1, 9-2, and 9-3 are further associated with each of processes A, B, and C, and managed.

[0145] 16 is a diagram for explaining an overview of the operation of the work analysis system 1 in embodiment 2. In the work analysis system 1 of this embodiment, the work analysis device 5 receives sensor data from each sensor 9 indicating the detection results of items 8 passing through the positions where each of processes A to C ends in the workplace 6, and calculates the number of items 8 passing through for each of processes A to C. The work analysis device 5 displays on the monitor 4 a production number graph indicating the number of items 8 produced by time, based on the number of items passing through each of processes A to C. FIG. 16 shows an example of a production number graph 75 for process A.

[0146] The production quantity graph 75 displays a production plan M1 indicating the target production quantity for each time in the workplace 6, and actual production results M2 indicating the production quantity calculated from the number of items 8 passing through each process A to C for each time. In the example of FIG. 16 , an abnormal state occurs in the period from time t11 to time t12, in which the production quantity of the actual production results M2 deviates from the production quantity of the production plan M1. The work analysis device 5 of this embodiment detects this period from time t11 to t12 as an abnormal period. The production plan quantity, such as the target production quantity for each time, is stored in advance in the storage unit 51a, for example, as normal state information D2 in this embodiment.

[0147] 2. Operation of the Work Analysis Device In the work analysis device 5 of this embodiment, for example, the control unit 50 functioning as the work detection unit 502 calculates the number of items 8 produced at each time based on sensor data from the sensor 9, thereby generating work information D1. For example, in step S1 of Fig. 6, in addition to image data for each frame, sensor data at the time the frame was captured may be acquired. The work information D1 stores, for example, the number of items produced at each time for each of processes A to C.

[0148] In this embodiment, the control unit 50 does not need to perform the image recognition process (S2 in FIG. 6) for detecting processes A to C based on the video data D0. Furthermore, the control unit 50 may record in the work information D1, for example, in step S3 in FIG. 6, the image capture time of each frame of the video data D0 in accordance with the sensor 9 that acquired the sensor data, in association with the process corresponding to the sensor 9.

[0149] 17 is a flowchart illustrating an example of the process of detecting an abnormal period in the work analysis device 5 of embodiment 2. The process of this flowchart is executed in the operation of the abnormality detection unit 503 and the video extraction unit 504 ( FIG. 7 ) instead of, for example, the process of step S12 of embodiment 1.

[0150] First, the control unit 50 acquires, for one process, the number of units produced at each time calculated from the detection result of the corresponding sensor 9, for example, from the work information D1 (S71).

[0151] Next, the control unit 50 sets a start time t0 and an end time t1 of a calculation period for calculating the throughput of the items 8 (S72). The end time t1 is set to, for example, a predetermined period n seconds (n is a positive number) after the start time t0.

[0152] The control unit 50 calculates the ratio of the number of units produced to the planned production number during the calculation period set in step S72, i.e., from time t0 to time t1, based on the number of units produced at each time acquired in step S71 and the normal status information D2 (S73).

[0153] The control unit 50 determines whether the ratio calculated in step S73 is less than a predetermined allowable ratio (S74). The predetermined allowable ratio is set from the perspective of detecting an abnormal state, such as when the production speed of the items 8 is slower than the production plan M1 and the number of units produced during the calculation period deviates from the production plan number.

[0154] If the calculated ratio is less than the predetermined allowable ratio (YES in S74), the control unit 50 determines that the process for which the ratio is calculated is an abnormal process, and stores abnormality information in the temporary memory unit 51b, which associates the calculation period and the camera number corresponding to the process based on the process information D3 (S76).

[0155] On the other hand, if the calculated ratio is not less than the predetermined allowable ratio (NO in S74), the control unit 50 increments the start time t0 and end time t1 of the calculation period (S77).

[0156] The control unit 50, for example, refers to the normal status information D2 and determines whether the incremented end time t1 is equal to or less than the end time of the work in the production plan at the workshop 6, i.e., whether it is within the production time (S78).

[0157] If time t1 is within the production time (YES in S78), the control unit 50 repeats the processing from step S73 onwards during the calculation period from the incremented time t0 to t1.

[0158] If time t1 is not within the production time (NO in S78), the control unit 50 determines, for example, whether the work information D1 includes the process next to the process for which the production number was obtained in step S71, i.e., whether there is a next process (S79).

[0159] If there is a next step (YES in S79), the control unit 50 executes the process from step S71 onwards for that step.

[0160] If there is no next step (NO in S79), the control unit 50 outputs the anomaly information stored in, for example, the temporary storage unit 51b from the anomaly detection unit 503 to the video extraction unit 504 (S80). Thereafter, the control unit 50 ends the processing of this flowchart.

[0161] According to the above processing, for each process, if the ratio of the production quantity to the production plan quantity for the calculation period is less than the allowable ratio (YES in S74), abnormality information for the process, etc. is retained (S76). This makes it possible to detect an abnormal state in which the production quantity deviates from the production plan quantity, such as a delay in the production of goods 8 in the workshop 6, and generate an extracted video as in the first embodiment. In the above processing, for example, after step S74, the control unit 50 may determine whether the difference between the cumulative production plan quantity and the production quantity from the start of the process to be calculated to time t1 is less than a predetermined allowable number, and if the difference is not less than the allowable number, retain the abnormality information.

[0162] FIG. 18 is a diagram illustrating the generation of an extracted video in the work analysis device 5 of embodiment 2. FIG. 18 shows an example in which work, such as the production of an item 8, is performed in the order of processes A, B, and C using materials input into equipment in a workplace 6. In the example of FIG. 18 , a workpiece processed in a previous process is further processed in a subsequent process. In such a scenario, if an abnormal condition occurs in one process, causing a delay in production, it is assumed that the cause lies in the process immediately preceding that process. Therefore, the work analysis device 5 of this embodiment generates an extracted video, for example, from a video of the process in which the abnormal condition occurred and a video of the process immediately preceding that process.

[0163] For example, if an abnormal condition is detected that causes a delay in production in process A, video from camera 2-1 corresponding to process A is extracted from cameras 2-1 to 2-3. This allows the video of process A, which is assumed to be the cause of the abnormal condition, to be analyzed. Furthermore, if production in process B is delayed, video from cameras 2-1 and 2-2 corresponding to processes A and B is extracted. In this case, it is assumed that production in process A, which outputs a workpiece to process B, is delayed, or that production in process B itself is delayed. With the extracted video in this embodiment, for example, user 3 can analyze the video of process B in addition to the video of process A, making it easier to analyze the work.

[0164] As described above, in this embodiment, the control unit 50 generates work information D1 based on sensor data obtained by the sensors 9 in the workplace 6 detecting, at each time point, items 8 (an example of a target object) on which processes A to C (an example of a plurality of tasks) are sequentially performed. The work information D1 includes information indicating the number of items 8 detected in the sensor data up to each time point. Generating work information D1 in this manner can reduce the processing load on the work analysis device 5 compared to generating work information D1 by analyzing, for example, video data captured by a camera. It can also reduce the cost of implementing the work analysis system 1.

[0165] In the present system 1, the worker W1 is not limited to a person, but may be a device that performs work in each process. In this case, the work information D1 may be generated based on log information from the device, instead of sensor data. In addition, in the present embodiment, an example has been described in which the abnormality period detection process ( FIG. 17 ) is executed instead of the process of step S12 in the first embodiment, but both the process of the flowchart in FIG. 17 and the process of step S12 may be executed.

[0166] (Other Embodiments) As described above, embodiments 1 and 2 have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to combine the components described in each of the above embodiments to create a new embodiment. Therefore, other embodiments will be described below as examples.

[0167] In the above embodiments, the work analysis device 5 accumulates video data D0 for the analysis period and then generates an extracted video. In this embodiment, the extracted video may be generated in real time while video data is acquired from each camera 2. The work analysis device 5 of this embodiment may store video data for a predetermined period shorter than the analysis period in, for example, the temporary storage unit 51b, and perform the operations of the video acquisition unit 501 and the work detection unit 502 (see FIG. 6 ) based on the video data. For example, instead of step S5 in FIG. 6 , it may determine whether acquisition of each frame for the predetermined period has ended, and repeat step S1 and subsequent steps until acquisition has ended.

[0168] In this embodiment, the work analysis device 5 may, for example, notify the anomaly detection unit 503 of a detected process change from the work detection unit 502, and execute the operations of the anomaly detection unit 503 and the video extraction unit 504 (see FIG. 7) based on the stored video data for each process. In this case, in the abnormal period detection process (S12 in FIG. 7), for example, instead of step S21 in FIG. 10, work information for the period of each process may be acquired, and the processes of steps S22 and S27 may not be executed. Also, for example, in step S13 in FIG. 7, a settings screen 71 (see FIG. 8) may be displayed so that the work time graph is updated sequentially.

[0169] In each of the above embodiments, an example has been described in which the work information D1 is generated based on either video data from the camera 2 or sensor data from the sensor 9. In the present embodiment, the work information D1 may be generated based on both video data and sensor data, and may include, for example, information indicating the work process performed by the worker W1 at each time and the number of items 8 produced at each time.

[0170] In the above-described first embodiment, an example was described in which the task analysis system 1 includes the sensor 9. In this embodiment, the task analysis system does not need to include the sensor 9. In this case, the task analysis system 1 can still operate in the same manner as the task analysis system 1 of the first embodiment, for example.

[0171] In the first embodiment described above, the work analysis device 5 extracts a predetermined period of N seconds each of videos of the process preceding and the process following the abnormal process in the extracted video generation process (S15) (S44, S45). In this embodiment, the period for extracting videos of the preceding and subsequent processes may differ depending on the type of abnormal condition in the abnormal process. For example, a priority may be set for each type of abnormal condition, and the higher the priority, the longer the video duration may be extracted. This makes it easier to perform detailed analysis of abnormal conditions with higher priorities from the extracted videos among multiple types of abnormal conditions. For example, an abnormal condition with a longer work period compared to the standard work period may be assigned a higher priority than an abnormal condition with a shorter work period.

[0172] In the first modification of the first embodiment described above, an example has been described in which, in the process of generating an extracted video, cameras 2 within a predetermined range from the camera position of the camera 2 corresponding to the abnormal process are identified as nearby cameras (S51). In this embodiment, for example, as in the above-described embodiment, a priority may be set for each type of abnormal state, and the higher the priority, the wider the range of cameras 2 identified as nearby cameras.

[0173] In the above-described first embodiment, an example has been described in which the control unit 50 of the work analysis device 5 serves as the work detection unit 502 and detects whether or not work is being performed for each of processes A to C. The control unit 50 as the work detection unit 502 is not limited to the above example, and may instead execute an image classification process to classify each frame image as showing which of processes A to C is shown in the image, or as showing none of the detection targets.

[0174] In the above embodiments, examples have been described in which, in the abnormal period detection process (S12), abnormal periods are detected from the entire analysis period based on the work information D1 for the analysis period. In this embodiment, a period in the analysis period during which abnormal periods are not detected may be set. For example, after step S22 in FIG. 10, the control unit 50 may determine whether the selected process falls within the set period, and if it does not fall within the set period, proceed to step S23 and subsequent processes, or if it does fall within the set period, proceed to step S27. This makes it possible to set periods such as break times in the workplace 6 so that they are not detected as abnormal periods, making analysis easier.

[0175] In the above embodiments, an example was described in which the work analysis system 1 includes three cameras 2-1 to 2-3. In the work analysis system of this embodiment, the number of cameras is not limited to three, and may be, for example, two or less or four or more. Furthermore, there does not have to be a one-to-one correspondence between each work process in the workplace 6 and each camera. For example, one camera may capture images of two or more processes. The process information D3 is not limited to the example in FIG. 4(B), and may be any information that associates the camera's capture range in the workplace 6 with the work area where each process is performed.

[0176] In the above embodiments, an example has been described in which each worker is identified when detecting processes performed in the workplace 6 (S2 in FIG. 6 ), for example, to enable analysis for each worker in the work analysis system 1. In this embodiment, analysis may be performed without distinguishing between multiple workers, and workers may not need to be identified when detecting processes, etc.

[0177] In each of the above embodiments, an example has been described in which the work analysis system 1 is applied to a workplace 6 having a conveyor line 61. The workplace 6 to which the work analysis system 1 is applied is not limited to the above examples, and the system may also be applied to a workplace where cell production is carried out, for example. Such a modified example will be described using FIG. 19 . FIG. 19 is a diagram illustrating a workplace to which the work analysis system 1 is applied in a further modified example of the first embodiment. In the workplace of this modified example, three workbenches 81, 82, and 83 are provided corresponding to work processes A, B, and C, respectively. For example, worker W1 sequentially performs processes A to C at the corresponding workbenches 81 to 83.

[0178] In the workplace of this modification, cameras 2-1 to 2-3 are positioned to capture the work areas of processes A to C at each workbench 81 to 83, and work analysis can be performed from the video captured by each camera 2, as in the above-described embodiments. In this case, for example, in the extracted video generation process ( FIG. 13 ) similar to that of Modification 1 of Embodiment 1, when process A is performed during an abnormal period, the extracted video may include a video of process C as a process performed near process A. In this modification, the work of processes A to C is not limited to being performed by a single worker W1, but may be performed by multiple workers. While FIG. 14 illustrates a workplace with one cell including workbenches 81 to 83, the present system 1 can also be applied to a workplace with multiple cells.

[0179] In each of the above embodiments, an example has been described in which the work analysis system 1 is applied to a workplace 6 such as a factory. In this embodiment, the workplace, i.e., the site, to which the work analysis system 1 and work analysis device 5 are applied is not limited to the above-mentioned workplace 6, but may be various other sites, such as a logistics warehouse or a store sales floor. Furthermore, the work assessed by the work analysis system 1 is not limited to the above-mentioned examples, but may be various types of work suited to various sites. For example, the system may be applied to the analysis of work that includes processes such as collecting and packing items 8 such as luggage at a logistics warehouse site.

[0180] In each of the above embodiments, an example was described in which the task analysis system 1 is applied to a person such as worker W1. In this embodiment, the worker to be analyzed by the task analysis system 1 may be a mobile object capable of performing various tasks, or may be production equipment or the like provided for each task. For example, the mobile object may be a robot, or various manned or unmanned vehicles.

[0181] As described above, the embodiments have been described as examples of the technology in the present disclosure, and for that purpose, the accompanying drawings and detailed description have been provided.

[0182] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0183] (Aspects of the Present Disclosure) The following describes examples of aspects of the present disclosure.

[0184] (Aspect 1) A task analysis device for analyzing video captured by one or more cameras of a workplace where multiple tasks are performed includes a memory unit that stores video data representing captured video of work areas in the workplace where each task is performed, and a control unit that detects, based on work information indicating the progress of the multiple tasks, an abnormal period in which the progress of each task deviates from a predetermined standard for each task among the multiple tasks. Based on the detected abnormal period, the control unit extracts, from the captured video represented by the video data, a first video of a first work area including a location where a first task was performed during the abnormal period, and a second video related to the first video. The second video includes at least one of video capture results of a second work area located near the first work area and video capture results of a second task performed at least either before or after the first task.

[0185] (Aspect 2) In the work analysis device according to aspect 1, the workplace is imaged by a plurality of cameras, and the memory stores work-related information associating each of the plurality of tasks with at least one of the plurality of cameras according to the work area of ​​each task. Based on the work-related information, the control unit extracts a first video from the videos captured by a camera of the plurality of cameras associated with a first task, and extracts a second video from the videos captured by a camera associated with at least one of a task performed in a second work area and a second task.

[0186] (Aspect 3) In the work analysis device according to aspect 1 or 2, the control unit generates a moving image that combines the first moving image and the second moving image based on the extracted moving image data representing the first moving image and the extracted moving image data representing the second moving image.

[0187] (Aspect 4) In the work analysis device according to any one of aspects 1 to 3, the second video includes a video in which a work area adjacent to the first work area in the workplace is captured as the second work area.

[0188] (Aspect 5) In the work analysis device according to any one of Aspects 1 to 4, the control unit extracts second videos based on movement line information indicating the location of the worker in the workplace at each time point, the second videos including videos capturing images of the location where the worker performed the second task at least one of the times before and after the first task.

[0189] (Aspect 6) In the work analysis device according to any one of Aspects 1 to 5, the control unit generates task information based on video data through image recognition processing, and the task information includes information indicating tasks performed at each time point in the workplace among the plurality of tasks.

[0190] (Aspect 7) In the work analysis device according to any one of Aspects 1 to 6, the control unit generates work information based on sensor data obtained by a sensor in a workplace detecting, at each time point, objects on which a plurality of tasks are performed sequentially, and the work information includes information indicating the number of objects detected in the sensor data up to each time point.

[0191] (Aspect 8) The work analysis device according to any one of Aspects 1 to 7 further includes an operation unit that inputs a user operation to set extraction conditions for extracting moving images from the captured moving images. The control unit extracts the first moving image and the second moving image from the captured moving images in response to the user operation input via the operation unit.

[0192] (Aspect 9) In the work analysis device according to any one of Aspects 1 to 8, the memory unit stores work standard information including, as the predetermined standard, at least one of the period required for each task by a worker, the order in which each task is performed among the plurality of tasks, and the time interval between two tasks among the plurality of tasks. The control unit detects an abnormal period by comparing the work information with the work standard information.

[0193] (Aspect 10) A task analysis method is provided for analyzing video captured by one or more cameras of a workplace where multiple tasks are performed. In this method, video data representing captured video of a work area in the workplace where each task is performed is stored in a computer storage unit, and a computer control unit detects, based on work information indicating the progress of the multiple tasks, an abnormal period in which the progress of each task deviates from a predetermined standard for each task among the multiple tasks. Based on the detected abnormal period, the control unit extracts, from the captured video represented by the video data, a first video of a first work area including a location where a first task was performed during the abnormal period, and a second video related to the first video. The second video includes at least one of an image of a second work area located near the first work area and an image of a second task performed at least either before or after the first task.

[0194] (Aspect 11) A program for causing a control unit of a computer to execute the task analysis method according to aspect 10 is provided.

[0195] The present disclosure is applicable to data analysis applications for analyzing the work of workers in various environments such as factories or logistics sites.

Claims

1. A work analysis device for analyzing video captured by one or more cameras of a workplace where multiple tasks are performed, comprising: a memory unit that stores video data indicating captured video of a work area in the workplace where each task is performed; and a control unit that detects, based on work information indicating the progress of the multiple tasks, an abnormal period in which the progress deviates from a predetermined standard for each task among the multiple tasks, wherein the control unit extracts, based on the detected abnormal period, from the captured video indicated by the video data, a first video in which a first work area including a location where a first task was performed during the abnormal period, and a second video related to the first video, wherein the second video includes at least one of an image capture result of a second work area located in the vicinity of the first work area, and an image capture result of a second task performed at least either before or after the first task.

2. The work analysis device described in claim 1, wherein the work area is imaged by a plurality of cameras, the memory unit stores work-related information that associates each of the plurality of tasks with at least one of the plurality of cameras according to the work area of ​​the each task, and the control unit extracts the first video from a video captured by a camera of the plurality of cameras associated with the first task based on the work-related information, and extracts the second video from a video captured by a camera associated with at least one of the task performed in the second work area and the second task.

3. The work analysis device of claim 1, wherein the control unit generates a video combining the first video and the second video based on the extracted video data representing the first video and video data representing the second video.

4. The work analysis device according to claim 1, wherein the second video includes a video in which a work area adjacent to the first work area in the workplace is captured as the second work area.

5. The work analysis device described in claim 1, wherein the control unit extracts the second video further based on movement line information indicating the position of the worker in the workplace at each time, and the second video includes a video capturing the position where the worker performed the second work at least one of the times before and after the first work.

6. The work analysis device according to claim 1, wherein the control unit generates the work information by image recognition processing based on the video data, and the work information includes information indicating which of the multiple works was performed at the workplace at each time.

7. The work analysis device according to claim 1, wherein the control unit generates the work information based on sensor data obtained by a sensor in the workplace detecting objects on which the plurality of tasks are performed sequentially at each time, and the work information includes information indicating the number of the objects detected in the sensor data up to each time.

8. The work analysis device of claim 1, further comprising an operation unit for inputting a user operation for setting extraction conditions for extracting videos from the captured video, wherein the control unit extracts the first video and the second video from the captured video in accordance with the user operation input at the operation unit.

9. The work analysis device according to claim 1, wherein the memory unit stores work standard information including at least one of the length of time each task is performed, the order in which each task is performed among the plurality of tasks, and the time interval between two tasks among the plurality of tasks as the specified standard, and the control unit detects the abnormal period by comparing the work information with the work standard information.

10. A work analysis method for analyzing video captured by one or more cameras of a workplace where a plurality of tasks are performed, comprising: storing video data indicative of captured video of a work area in the workplace where each task is performed in a memory unit of a computer; a control unit of the computer detects, based on work information indicative of a progress status of the plurality of tasks, an abnormal period in which the progress status deviates from a predetermined standard for each task among the plurality of tasks; and extracting, based on the detected abnormal period, a first video in which a first work area including a location where a first task was performed during the abnormal period and a second video related to the first video from the captured video indicated by the video data; and the second video including at least one of an image capture result of a second work area located in the vicinity of the first work area, and an image capture result of a second task performed at least either before or after the first task.

11. A program for causing a control unit of a computer to execute the task analysis method according to claim 10.

Citation Information

Patent Citations

  • Process monitoring apparatus, and control method and program for process monitoring apparatus

    JP2018128729A

  • Worker management system

    JP2022067694A

  • Work management device, work management method, and work management program

    JP2022113099A