Program, information processing method, and information processing apparatus

Abstract

The present invention provides a program or similar tool that can determine whether or not the tasks performed by the monitored entity include any actions that are not normal. [Solution] The computer acquires video footage of the workspace where the monitored object is working, using a camera attached to the monitored object, for each of the monitored objects during their work. The computer inputs each of the acquired video footage into a trained model that has been trained to detect whether or not there is abnormal operation in the work performed by the object when video footage of the workspace in which the object is working is input, and obtains the detection result for the presence or absence of abnormal operation for each monitored object. If the computer obtains a detection result for abnormal operation, it outputs an alert to the terminal of the administrator who manages the monitored object.

Description

Technical Field

[0001] This disclosure relates to a program, an information processing method, and an information processing apparatus.

Background Art

[0002] Patent Document 1 discloses a technique for calculating the time required for a task (task time) based on image data obtained by photographing tasks performed in a preset order in a production line, and monitoring the task state based on the task time. In the technique disclosed in Patent Document 1, when the task time calculated based on the image data is outside the normal range, it is determined that an abnormality has occurred in the task time.

Prior Art Documents

Patent Documents

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique disclosed in Patent Document 1, when the task time is outside the normal range, it is determined that a task different from the normal task has been performed. However, even a task performed within the normal range of the task time is not necessarily a normal task. Therefore, it is difficult to determine whether a normal task has been performed only by determining whether the task time is within the normal range.

[0005] An object of this disclosure is to provide a program or the like capable of determining whether an operation that is not a normal operation is included in the operation performed by the monitoring target.

Means for Solving the Problems

[0006] A program according to one aspect of this disclosure acquires video footage of the workspace where multiple monitored objects are working, captured by a camera attached to the monitored object, during the work of each of the monitored objects. The program inputs each of the acquired video footage of the monitored objects into a trained model that has been trained to detect whether or not there is abnormal operation in the work performed by each object when video footage of the workspace where the object is working is input, and obtains a detection result for the presence or absence of abnormal operation for each monitored object. If a detection result for abnormal operation is obtained, the program causes a computer to execute a process that outputs an alert to the terminal of the administrator who manages the monitored objects. [Effects of the Invention]

[0007] In one aspect of this disclosure, it is possible to determine whether the work performed by the monitored entity includes any operations that are not normal. [Brief explanation of the drawing]

[0008] [Figure 1] This is an explanatory diagram showing an example of a video to be processed. [Figure 2] This is an explanatory diagram showing an example of the configuration of an information processing system. [Figure 3] This is a block diagram showing an example of the configuration of an information processing system. [Figure 4] This is an explanatory diagram showing an example of a learning model configuration. [Figure 5] This is an explanatory diagram showing an example of a record layout for a workline database. [Figure 6] This is an explanatory diagram showing an example of a record layout in a factory database. [Figure 7] This is a flowchart illustrating an example of the training process for a learning model. [Figure 8] This is a flowchart illustrating an example of the training process for a learning model. [Figure 9] This is an explanatory diagram showing an example screen related to the learning process. [Figure 10] This is an explanatory diagram showing an example screen related to the learning process. [Figure 11]This flowchart shows an example of a procedure for detecting abnormal operation. [Figure 12] This flowchart shows an example of a procedure for checking for abnormal operation. [Figure 13] This is an explanatory diagram showing an example screen related to the confirmation process. [Figure 14] This is an explanatory diagram showing an example screen related to the confirmation process. [Figure 15] A flowchart illustrating an example of the procedure for presenting work results. [Figure 16] This is an explanatory diagram showing an example screen related to the processing of displaying work results. [Modes for carrying out the invention]

[0009] The program, information processing method, and information processing apparatus of this disclosure will be described in detail below with reference to the drawings illustrating their embodiments.

[0010] (Embodiment 1) This embodiment describes an information processing system that detects whether a worker (the person being monitored) is performing normal operations based on video footage (monitored video) of the workspace where the worker is performing their work, captured using a camera attached to the worker. In this embodiment, the information processing system allows the worker to determine whether they are performing normal operations in real time while they are working, based on the captured video footage. In this embodiment, the worker is, for example, a worker performing work in a manufacturing site such as a factory, and the work performed by the worker includes various tasks necessary for manufacturing products, such as assembling parts or products, and inspecting parts or products. Note that the work in this embodiment is not limited to tasks performed in the manufacturing of products, as long as a series of actions are performed in a predetermined order. In this embodiment, normal operation means standard operation that should be used as a model, and actions that differ from normal operation, or actions that deviate from the range that can be judged as normal operation (abnormal operation), are described as "abnormal operation." That is, a state in which the worker is not performing normal operations is expressed as "performing abnormal operations." In this embodiment, a series of operations in which all operations are normal (normal operation) may be described as a normal cycle, and an operation that includes at least some abnormal operations may be described as an abnormal cycle.

[0011] FIG. 1 is an explanatory diagram showing an example of a video to be processed. The video to be processed in this embodiment is video data including a plurality of images (still images), such as 30 images or 15 images per second, and is a captured image (captured video) that captures the area near the hands of an operator performing work as shown in FIG. 1A and the work space where the operator is working. Note that the video to be processed is a captured video captured using a capturing device worn by an operator, such as a wearable camera, but may also be a captured video captured using a capturing device fixed at a position where the work space and the area near the hands of the operator can be captured as shown in FIG. 1A. The wearable camera is, for example, an ear-mounted type, a headband type (headstrap type), a glasses type camera, etc., and is worn and used on the head (face) of an operator. Also, the wearable camera may be configured to be worn by an operator by attaching it to the temple of the glasses, the brim of a hat, or a helmet worn by the operator, or may be configured to be worn on the clothing or body of the operator using a clip or a chest harness, etc. Furthermore, the wearable camera may be configured by attaching a portable terminal, such as a smartphone, a tablet terminal, or a portable game machine having a camera, to a mounting belt or the like for mounting on the head or chest of a user.

[0012] The video to be processed in this embodiment is not only a video (hereinafter referred to as a cycle video) captured while an operator performs a series of operations, but may also be a video including a plurality of cycle videos as shown in FIG. 1B. The example in FIG. 1B is a video that captures how an operator repeatedly performs operation 1 (assembly of case A). Note that the video to be processed may be a video that captures how an operator continuously performs different operations in addition to a video that captures how an operator continuously performs the same operation. Different operations are, for example, a plurality of operations required when assembling one product. Also, in the cycle videos included in one video, the operator performing each operation may be the same operator or different operators.

[0013] Figure 2 is an explanatory diagram showing an example configuration of an information processing system. The information processing system of this embodiment includes an anomaly detection device 10, a camera 20, a notification terminal 30, a supervisor terminal 40, and a server 50, and each device is communicated with the others via a network N. The network N may be the Internet or a public telephone network, or it may be a LAN (Local Area Network) built within a facility such as a factory where the information processing system is installed. The camera 20 is a camera used by workers when they are performing their work, and is a so-called wearable camera. The camera 20 has a shooting unit and a communication unit, and transmits image data obtained by shooting with the shooting unit to the anomaly detection device 10 via the communication unit. The camera 20 and the anomaly detection device 10 are configured to perform wireless communication, for example, and the communication unit of the camera 20 has a communication module for performing wireless communication processing compliant with Bluetooth®, for example. The camera 20 and the anomaly detection device 10 may also be configured to perform wired communication via a cable such as a USB (Universal Serial Bus) cable, in which case the communication unit of the camera 20 may have a communication module for performing wired communication processing. Furthermore, the camera 20 and the anomaly detection device 10 may be configured to communicate via a network N, in which case the communication unit of the camera 20 may have a communication module for connecting to the network N. When the camera 20 is powered on and starts operating, it starts the shooting process by the shooting unit and the communication process by the communication unit. The camera 20 may transmit the captured images to the anomaly detection device 10 in real time each time an image is acquired by the shooting unit, or it may temporarily store the captured images in a storage unit and transmit them to the anomaly detection device 10 in batches at predetermined intervals (for example, every 10 seconds, every 30 seconds, every minute, etc.).

[0014] The abnormality detection device 10 is an information processing device capable of various information processing and information transmission and reception, such as a personal computer, a server computer, a workstation, a tablet terminal, etc. In the example of FIG. 2, one abnormality detection device 10 is configured to communicate with one camera 20, and a plurality of pairs of one camera 20 and one abnormality detection device 10 are provided, but it is not limited to this configuration. For example, one abnormality detection device 10 may be provided, and one abnormality detection device 10 may communicate with all the cameras 20 and acquire captured images from each camera 20. Also, a plurality of sets may be provided with one abnormality detection device 10 and a plurality of cameras 20 as one set.

[0015] The notification terminal 30 and the supervisor terminal 40 are terminals used by supervisors who supervise workers. The supervisor (administrator) includes, in addition to supervisors such as line chiefs, line leaders, and line managers who manage and supervise workers working on a work line such as an assembly line or a production line, assistants who assist the supervisors. The notification terminal 30 is a wearable device such as a smartwatch and has a communication unit and a notification unit, etc. The communication unit has a communication module for connecting to the network N by wireless communication. The notification unit includes one or more of a display unit, a speaker, a lamp, a buzzer, a vibrator, etc., and notifies the supervisor carrying the notification terminal 30 of the occurrence of a predetermined situation by notification processing such as displaying a message on the display unit, outputting a message by voice from the speaker, lighting or flashing the lamp, sounding the buzzer, vibrating by the vibrator, etc. In the present embodiment, when the abnormality detection device 10 detects that a worker is not performing a normal operation (performing an abnormal operation), a predetermined signal is transmitted to the notification terminal 30, and when the notification terminal 30 receives the predetermined signal from the abnormality detection device 10, the notification terminal 30 executes notification processing by the notification unit. Note that the abnormality detection device 10 may transmit a signal corresponding to the type of detected abnormal operation to the notification terminal 30, and the notification terminal 30 may be configured to perform different notification processing according to the received signal. The supervisor terminal 40 is an information processing device capable of various information processing and information transmission and reception, such as a personal computer, a tablet terminal, a smartphone, etc.

[0016] Server 50 is an information processing device capable of various information processing and information transmission / reception, such as a personal computer, server computer, or workstation. Server 50 may be a server managed by a company that owns a factory or the like where an information processing system is built, or it may be a server managed by a company that uses an information processing system to support the monitoring of workers' work status.

[0017] In this embodiment of the information processing system, one anomaly detection device 10 and multiple cameras 20 are provided for each work line, and multiple cameras are attached to multiple workers performing work on that work line. That is, a set including one anomaly detection device 10 and multiple cameras 20 is provided for each work line in the factory. In such a system, the cameras 20 photograph the work space while the worker is working and transmit the captured images to the anomaly detection device 10 in real time. The anomaly detection device 10 acquires the captured images from each camera 20 and determines whether or not the work performed by the worker includes an abnormal operation based on the acquired captured images. If it determines that an abnormal operation is included, it notifies the supervisor of the work line of the occurrence of the abnormal operation via a notification terminal 30. The supervisor who is notified of the occurrence of the abnormal operation via the notification terminal 30 uses a supervisor terminal 40 to check the captured images that the anomaly detection device 10 has determined to be abnormal, and determines whether or not it was truly an abnormal operation. If it is determined that it was truly an abnormal operation, the supervisor provides feedback and guidance to the worker who performed the abnormal operation. This allows for real-time determination of whether or not an abnormal operation is included in the worker's work, and enables early correction of the worker's work (operation) if an abnormal operation occurs, contributing to improved worker efficiency and product quality.

[0018] Figure 3 is a block diagram showing an example configuration of an information processing system. Figure 3 shows an example configuration of an anomaly detection device 10, a supervisor terminal 40, and a server 50, while the detailed configuration of the camera 20 and notification terminal 30 is omitted. The anomaly detection device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, a wireless communication unit 17, etc., and each of these units is interconnected via a bus. The control unit 11 includes one or more processors (arithmetic processing units) such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), or an AI chip (AI semiconductor). The control unit 11 executes the processing that the anomaly detection device 10 should perform by reading and executing the program 12P stored in the storage unit 12. Note that if the control unit 11 includes multiple processors, each process may be executed by the same processor, or each process may be executed by different processors.

[0019] The memory unit 12 includes RAM (Random Access Memory), flash memory, hard disk, SSD (Solid State Drive), etc. The memory unit 12 stores the program 12P (program product, computer program) executed by the control unit 11 and various data. The memory unit 12 also temporarily stores data generated when the control unit 11 executes the program 12P. The memory unit 12 also stores the learning model 12M that has been trained by machine learning. The learning model 12M is intended to be used as a program module that constitutes artificial intelligence software. The learning model 12M performs predetermined calculations on input values ​​and outputs the calculation results. The memory unit 12 stores information that defines the learning model 12M, such as information on the layers of the learning model 12M, information on the nodes that constitute each layer, and weights (connection coefficients) between nodes. The learning model 12M is prepared for each type of work (work content) performed by the worker. In addition to being prepared for each type of work, the learning model 12M may also be prepared for each workspace where the same work is performed, or for each factory or base where the same work is performed. Furthermore, the storage unit 12 stores the work line DB 12a. The storage unit 12 may be composed of multiple storage devices, and a part of the storage unit 12 may be other storage devices connected to the anomaly detection device 10, or other storage devices that the anomaly detection device 10 can communicate with.

[0020] The communication unit 13 has a communication module for processing wired or wireless communication and transmits and receives information with other devices via the network N. The wireless communication unit 17 has a communication module for processing wireless communication and transmits and receives information directly with, for example, the camera 20. The input unit 14 receives operation input from the user and sends control signals corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display or an organic EL display, etc., and displays various information according to instructions from the control unit 11. Part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.

[0021] The reading unit 16 reads information recorded on the recording medium 10a, such as a memory card or optical disc. The program 12P and various data to be stored in the storage unit 12 may be read by the control unit 11 from the recording medium 10a via the reading unit 16 and stored in the storage unit 12. Alternatively, the program 12P and various data may be written to the storage unit 12 during the manufacturing or installation stage of the anomaly detection device 10, or the control unit 11 may download them from other devices via the communication unit 13 and store them in the storage unit 12.

[0022] The anomaly detection device 10 is not limited to a single computer; it may be a multi-computer system comprising multiple computers, or it may be a virtual machine virtually constructed within a single device by software. Furthermore, the anomaly detection device 10 may be a local server installed within the facility where the anomaly detection device 10 (information processing system) is located, or it may be a cloud server connected via network N. In the following description, the anomaly detection device 10 will be described as a single computer. In addition, the program 12P may be deployed and executed on a single computer or at a single site, or it can be distributed across multiple sites and deployed to run on multiple computers interconnected via network N. Moreover, the input unit 14 and display unit 15 of the anomaly detection device 10 are not mandatory; it may be configured to accept operations via a connected computer, or it may be configured to output the information to be displayed to an external display device.

[0023] The supervisor terminal 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, a display unit 45, etc., and these units are interconnected via a bus. The control unit 41, storage unit 42, communication unit 43, input unit 44, and display unit 45 of the supervisor terminal 40 have the same configuration as the control unit 11, storage unit 12, communication unit 13, input unit 14, and display unit 15 of the anomaly detection device 10, so their description is omitted. The server 50 includes a control unit 51, a storage unit 52, a communication unit 53, etc., and these units are interconnected via a bus. The control unit 51, storage unit 52, and communication unit 53 of the server 50 have the same configuration as the control unit 11, storage unit 12, and communication unit 13 of the anomaly detection device 10, so their description is omitted. The storage unit 52 of the server 50 stores the factory DB 52a in addition to the program 52P executed by the control unit 51.

[0024] In this embodiment, the anomaly detection device 10 may have the same functions as the server 50 and perform the processing that the server 50 executes. For example, each of the multiple anomaly detection devices 10 may perform the processing that the server 50 executes, or the server 50 and the multiple anomaly detection devices 10 may be implemented in a single device.

[0025] Figure 4 is an explanatory diagram showing an example configuration of the learning model 12M. Figure 4A is an explanatory diagram of the learning process of the learning model 12M, and Figure 4B is an explanatory diagram of the estimation process using the learning model 12M. The learning model 12M is constructed using algorithms such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), CNN (Convolutional Neural Network), and Transformer, and may also be constructed by combining multiple algorithms. The learning model 12M is a model that has been pre-trained on videos of humans performing a wide variety of actions (videos including human movements), and is generated by loading training cycle videos. For the training cycle videos, videos in which each action included in a series of tasks is a standard action (normal action) to be used as a model (hereinafter referred to as a normal cycle video) are used.

[0026] During the learning process, the learning model 12M extracts and stores features from the input training cycle video (normal cycle video). For example, the learning model 12M detects moving objects in the normal cycle video and extracts features in the area where the moving objects are captured. For example, the learning model 12M extracts features for each unit time in the area where the moving objects are captured. Then, when the video to be estimated is input to the estimation process, the learning model 12M, which has already loaded the normal cycle video, extracts a cycle video from the video to be estimated that performs the same operation as the normal cycle video, based on the image features of the normal cycle video extracted during the learning process. For example, the learning model 12M extracts features in the area where moving objects are captured in the video to be estimated, and based on the image features of the normal cycle video and the video to be estimated, it identifies parts of the video to be estimated that are similar to and different from the normal cycle video. Sections similar to (or not similar to) a normal cycle video can be defined as sections where the similarity between the image features of the normal cycle video and the image features of the target video is above a threshold (or below a threshold), and this threshold may be changeable by the user. Based on the sections similar to and not similar to the normal cycle video identified for the target video, the learning model 12M divides the target video into cycle videos of the same operation as the normal cycle video. The learning model 12M also identifies sections that are not similar to the normal cycle video and their time zones (differences from the normal cycle video) in each divided cycle video. When the learning model 12M identifies sections that are not similar to the normal cycle video and their time zones, it identifies the time zones in the normal cycle video corresponding to those sections and outputs this information as information indicating the differences between the normal cycle video and the target cycle video. In this embodiment, in the target cycle video, the differences from the normal cycle video are determined to be sections of operation different from normal operation, i.e., sections of abnormal operation.

[0027] The learning model 12M of this embodiment is configured to detect the following five types of abnormal operation locations as differences between the cycle video to be estimated and the normal cycle video. Note that the differences from the normal cycle video are not limited to the following examples. 1. Differences from normal operation 2. Areas similar to abnormal operation 3. Locations where the operating time exceeded or fell below a predetermined value compared to the operating time required for normal operation (locations with long or short operating times), and cycles where the cycle time exceeded or fell below a predetermined value compared to the cycle time required for a normal cycle (locations with long or short cycle times). 4. A location where any of the operations included in a normal cycle are missing (missing location) 5. Locations where the order of operations included in a normal cycle has been changed.

[0028] Specifically, the learning model 12M detects locations that are "different from normal operation" where the similarity to each operation in the normal cycle video (normal operation), or an operation loaded into the learning model 12M as normal operation, is below a threshold. It outputs the time period (start date and time and end date and time) of the detected location and information indicating that the operation during that time period is different from the normal cycle. The time period of the abnormal operation is represented by the elapsed time from the start of playback of the normal cycle video. The learning model 12M also detects locations that are "similar to abnormal operation" where the similarity to an operation loaded into the learning model 12M as abnormal operation is above a threshold. It outputs the time period of the detected location and information indicating that the operation during that time period is similar to abnormal operation. Here again, the time period of the detected location is represented by the elapsed time from the start of playback of the normal cycle video. Furthermore, the learning model 12M detects "locations with long or short operating times" where the similarity to normal operation is above a threshold, and the duration of that location in the estimated cycle video exceeds or falls below the normal operation time by a predetermined value or more. It outputs the time period of the detected location and information indicating whether the operating time for that time period is long or short. Similarly, the learning model 12M detects "locations with long or short cycle times" where the similarity to normal cycle video is above a threshold, and the duration of that cycle in the estimated cycle video exceeds or falls below the normal cycle time by a predetermined value or more. It outputs the time period of the detected cycle and information indicating whether the cycle time for that time period is long or short. In addition, the learning model 12M detects "locations with missing operations" where operations are missing from the normal cycle video, and outputs the time period of the detected operation and information indicating that the operation for that time period is missing. Furthermore, the learning model 12M detects "locations where actions have been swapped" by identifying actions in a normal cycle video whose order has been reversed, and outputs the time period of the detected action and information indicating that the actions in that time period have been swapped. The threshold and predetermined values ​​used by the learning model 12M to determine a location is a difference from a normal cycle video can be changed, and by setting them to appropriate values, different learning models 12M can be generated for each type of work, factory, or location.

[0029] Therefore, the learning model 12M is configured to take the video to be estimated as input, perform calculations to extract cycle videos from the input video, and perform calculations to detect the differences between each extracted cycle video and a normal cycle video, and output the results of these calculations. Specifically, the learning model 12M has an input layer into which the video to be estimated is input, an intermediate layer that extracts image features from the input video and performs various calculations, and an output layer that outputs the start and end times of each cycle video extracted from the video based on the calculation results of the intermediate layer, the start and end times of the differences between each cycle video and a normal cycle video, and information indicating the type of abnormal operation. The start and end times of the cycle videos are represented by the elapsed time from the start of playback of the video, and the start and end times of the differences between each cycle video and a normal cycle video are represented by the elapsed time from the start of playback of the normal cycle video.

[0030] By using such a learning model 12M, when the anomaly detection device 10 inputs a video to be estimated into the learning model 12M, it can extract cycle videos from the video to be estimated based on the output information from the learning model 12M (start and end times of cycle videos, start and end times of the difference between each cycle video and a normal cycle video, and information indicating the type of abnormal operation), and can also identify the difference between each cycle video and a normal cycle video. Furthermore, when multiple training normal cycle videos are loaded into the learning model 12M, the learning model 12M calculates the average value of the image features per unit time in the multiple loaded normal cycle videos, and uses the calculated average value as the image features of the normal cycle videos for subsequent estimation processing.

[0031] In this embodiment, the anomaly detection device 10 identifies the difference between each cycle video and a normal cycle video based on the output information from the learning model 12M, and also identifies the type of abnormal operation at the identified difference. In this embodiment, the anomaly detection device 10 identifies which of the five types of abnormal operation the abnormal operation detected by the learning model 12M is.

[0032] The learning model 12M can be trained (by reading normal cycle videos for training) by the anomaly detection device 10, the server 50, or other learning devices. In this embodiment, for example, training may be performed on the server 50, and the trained learning model 12M may be downloaded from the server 50 to each anomaly detection device 10 via the network N or recording medium 10a and stored in the storage unit 12. Furthermore, the learning model 12M is not limited to a configuration in which videos of the worker's hands and workspace are input. For example, the anomaly detection device 10 may be configured to estimate the skeleton of the worker's fingers based on each frame (each image) contained in the video, generate finger skeleton data, and input the time-series finger skeleton data to the learning model 12M. In this case, the control unit 11 of the anomaly detection device 10 uses a technique to extract the joint positions of a person in an image, such as OpenPose, to estimate the skeleton of the worker's fingers in each frame and extract the joint positions of the worker's fingers. The control unit 11 may obtain time-series skeletal data of the fingers by performing a process to extract the joint positions of the fingers for each frame included in the video, and use the obtained time-series skeletal data of the fingers as input to the learning model 12M.

[0033] Figure 5 is an explanatory diagram showing an example of the record layout of the work line DB 12a. The work line DB 12a stores information about each work line installed in the factory (location) where the anomaly detection device 10 is installed, and information about the workers performing work in the workspace of each work line. The work line DB 12a shown in Figure 5 includes a line ID column, a line information column, a supervisor information column, a notification terminal information column, a workspace information column, a worker ID column, a camera information column, a video information column, etc., and stores each piece of information in association with identification information (line ID) that identifies the work line. The line information column stores information about the work line, such as the location of the work line in the factory, information about the products being manufactured on the work line, and information about the work being performed on the work line. The supervisor information column stores information about the person in charge of the work line (supervisor), such as the supervisor's identification information (ID), name, department, etc. The notification terminal information column stores information about the notification terminal 30 carried by the supervisor, such as the identification information (ID) of the notification terminal 30, terminal information related to the notification terminal 30, and destination information for communicating with the notification terminal 30. The workspace information column stores information about the workspace provided on the work line, such as the identification information (ID) of the workspace and the location of the workspace within the factory. The worker ID column stores the identification information (worker ID) of the worker using the workspace, associated with the workspace ID. The camera information column stores information about the camera 20 used by the worker, associated with the worker ID, such as the identification information (ID) of the camera 20, equipment information related to the camera 20, and destination information for communicating with the camera 20. The video information column stores information about the video (monitored video) captured by the camera 20, associated with the worker ID and camera ID. The video information includes the start and end dates and times of recording, the file name of the video data stored in the storage unit 12, the cycle time (start and end dates and times) of each cycle video obtained by dividing the video using the learning model 12M, and the detection result of the difference between each cycle video and the normal cycle video. Videos of the monitored worker are stored in the storage unit 12, and the file name of the video data includes the folder name and file name for reading from the storage unit 12. The contents of the work line DB 12a are not limited to the example shown in Figure 5.

[0034] Figure 6 is an explanatory diagram showing an example of the record layout of the factory DB52a. The factory DB52a stores information about workers performing tasks at each factory (location). The factory DB52a shown in Figure 6 includes columns such as factory ID, worker ID, workspace ID, work information, date, total cycle count, and abnormal cycle information, and stores worker information associated with identification information (factory ID) that identifies the factory. Worker information is stored associated with the worker ID. The work information column stores information about the tasks performed by the worker on the work line (workspace). The total cycle count and abnormal cycle information columns are associated with the date and store, for example, the number of tasks performed in a day (total cycle count) and information about abnormal cycles (abnormal cycle information) that were determined to include abnormal operation from videos taken during the day's work. The abnormal cycle information includes the file name of the video data stored in the storage unit 12, the cycle time (start date and time and end date and time) of each cycle video obtained by dividing the video using the learning model 12M, and the detection result of the difference between each cycle video and a normal cycle video. The contents stored in the factory DB 52a are not limited to the example shown in Figure 6, and for videos determined to be normal, the file name of the video data, the cycle time of each cycle video obtained by dividing the video, and the detection result of the difference between each cycle video and a normal cycle video (e.g., no difference, normal) may also be stored.

[0035] The following describes the learning process in which a normal cycle video is loaded into the learning model 12M in the information processing system of this embodiment. In this embodiment, the cycle video used in the learning process includes not only a normal cycle video in which all of the operations are normal operations, but also cycle videos in which some operations are set to be normal operations, cycle videos in which some operations are set to be abnormal operations, cycle videos that include operations set to be normal operations and operations set to be abnormal operations, etc. Hereinafter, the parts (time periods) set to be normal operations or abnormal operations will be referred to as partial videos.

[0036] Figures 7 and 8 are flowcharts showing an example of the learning process procedure for the learning model 12M, and Figures 9 and 10 are explanatory diagrams showing example screens related to the learning process. In Figures 7 and 8, the left side shows the processing performed by the configuration terminal used to set the cycle video used for the learning process, and the right side shows the processing performed by the learning device. In this embodiment, the configuration terminal can be a supervisor terminal 40, an anomaly detection device 10, a terminal used by the user performing the learning process, etc., and the learning device can be a server 50, etc. In the following, we will explain using the case in which the supervisor terminal 40 is used as the configuration terminal and the server 50 is used as the learning device as an example.

[0037] The control unit 41 of the supervisor terminal 40 displays a setting screen on the display unit 45 that accepts the setting (specification) of a cycle video to be used for the learning process (S11). The cycle video to be used for the learning process is a video that has been filmed (recorded) in advance. The setting screen is, for example, generated in advance and stored in the storage unit 52 of the server 50, and the control unit 41 obtains the setting screen from the server 50 and displays it on the display unit 45. The setting screen has an input field that accepts the specification (selection) of any video from the videos stored in the storage unit 42 and videos that can be obtained from other devices that the supervisor terminal 40 can communicate with, and the user specifies any video via the input field. The video here may be a video that includes one cycle video (normal cycle video) to be used for the learning process, or it may be a video that includes multiple cycle videos, including a cycle video to be used for the learning process. The control unit 41 of the supervisor terminal 40 determines whether or not it has accepted the specification of a video via the setting screen (S12), and if it determines that it has not been accepted (S12: NO), it waits until it is accepted.

[0038] If the control unit 41 determines that it has received a video selection request (S12: YES), it displays the selected video on the settings screen (S13). Figure 9 shows an example of the settings screen, which has a video area R1 for displaying the selected video and a seek bar R2 for indicating the playback position of the displayed video. The user finds the section of the displayed video that shows a series of normal operations (normal cycle video) and specifies the start and end positions of the normal cycle video on the seek bar R2. In Figure 9, two normal cycle videos are selected, but it is sufficient to select one or more normal cycle videos. If the user has selected a normal cycle video and wants to use the selected normal cycle video (normal video) for the learning process, they operate the "Save as Normal Cycle" button.

[0039] The control unit 41 determines whether or not it has received a request for a cycle video (normal cycle video) via the seek bar R2 (S14). If it determines that it has not received the request (S14: NO), it waits until it does. If the control unit 41 determines that it has received a request for a cycle video (S14: YES), it determines whether or not the "Save as Normal Cycle" button has been pressed (S15). If the control unit 41 determines that the "Save as Normal Cycle" button has been pressed (S15: YES), it extracts the specified cycle video (normal cycle video) from the currently displayed video (S16). The extracted normal cycle video is stored in the storage unit 42. The control unit 41 sends the extracted normal cycle video to the server 50 (learning device) (S17).

[0040] The control unit 51 of the server 50 acquires a normal cycle video from the supervisor terminal 40 (S18) and stores the acquired normal cycle video in the storage unit 42. The control unit 51 then executes a learning process using the acquired normal cycle video (S19). In the learning process, the control unit 51 inputs the normal cycle video to the learning model 12M, causes the learning model 12M to detect moving objects in the normal cycle video, extracts image features for each unit of time in the area where the moving objects are captured, and stores them.

[0041] If the control unit 41 determines that the "Save as Normal Cycle" button has not been pressed (S15: NO), it proceeds to step S20. In addition to using the cycle video specified on the seek bar R2 as the normal cycle video, the user can also specify a part of the cycle video specified on the seek bar R2 (partial video) as a video of normal operation or abnormal operation to be used as a learning target. Therefore, the user finds the part of the cycle video specified on the seek bar R2 that captures normal operation or abnormal operation (partial video), and specifies the start and end positions of the partial video on the seek bar R2. In Figure 10A, the partial video from 1:32 to 1:50 is specified. Note that if a partial video is specified on the seek bar R2, the first image of the specified partial video may be displayed in the video area R1. When a partial video is specified, the user presses the "Save as Normal" button if the specified partial video is a part of normal operation, and presses the "Save as Abnormal" button if the specified partial video is a part of abnormal operation.

[0042] The control unit 41 determines whether or not it has received a request for a partial video via the seek bar R2 (S20). If it determines that it has not received the request (S20: NO), it waits until it does. If the control unit 41 determines that it has received a request for a partial video (S20: YES), it determines whether or not the "Save as Normal" button has been pressed (S21). If the control unit 41 determines that the "Save as Normal" button has been pressed (S21: YES), it extracts the specified cycle video from the currently displayed video (S22), and then extracts the specified partial video (a portion of the video in normal operation) from the extracted cycle video (S23). The extracted cycle video and partial video are stored in the storage unit 42. The control unit 41 sends the extracted cycle video and partial video to the server 50 (S24).

[0043] The control unit 51 of the server 50 acquires cycle videos and partial videos of normal operation from the supervisor terminal 40 (S25), and stores the acquired cycle videos and partial videos in the storage unit 52. The control unit 51 then performs a learning process using the acquired cycle videos and partial videos (S26). Here, the control unit 51 inputs the cycle videos and partial videos of normal operation into the learning model 12M, causes the learning model 12M to detect moving objects in the cycle videos and partial videos, extracts image features for each unit time in the area where the moving objects are captured, and stores them. The learning model 12M also adds information indicating that the partial video is in normal operation to the image features and stores them.

[0044] If the control unit 41 determines that the "Save as normal" button has not been pressed (S21: NO), it determines whether the "Save as abnormal" button has been pressed (S27). If the control unit 41 determines that the "Save as abnormal" button has been pressed (S27: YES), it extracts the specified cycle video from the currently displayed video (S28), and then extracts the specified partial video (partial video of abnormal operation) from the extracted cycle video (S29). The extracted cycle video and partial video are stored in the storage unit 42. The control unit 41 sends the extracted cycle video and partial video to the server 50 (S30).

[0045] The control unit 51 of the server 50 acquires the cycle video and partial video of abnormal operation from the supervisor terminal 40 (S31), and stores the acquired cycle video and partial video in the storage unit 52. The control unit 51 then performs a learning process using the acquired cycle video and partial video (S32). Here, the control unit 51 inputs the cycle video and partial video of abnormal operation into the learning model 12M, causes the learning model 12M to detect moving objects in the cycle video and partial video, extracts image features for each unit time in the area where the moving object is captured, and stores them. The learning model 12M also adds information indicating that it is an abnormal operation to the image features of the partial video and stores them. The control unit 41 terminates the process if it determines that the "Save as Abnormal" button has not been operated (S27: NO), for example, if the cancel button provided on the settings screen is operated.

[0046] Figure 10B shows another example of a settings screen that accepts the specification of partial videos of normal operation and partial videos of abnormal operation. In the screen of Figure 10B, the specified cycle video is displayed in the video area R1, and the seek bar R2 indicates the playback position of the displayed cycle video. The screen of Figure 10B also has a partial video area R3 that displays the partial video specified on the seek bar R2, and normal and abnormal buttons associated with each partial video to specify whether each partial video is a shooting location of normal operation or an abnormal operation. The user finds the shooting locations of normal operation and / or abnormal operation (partial videos) from the displayed cycle video and specifies the start and end positions of the partial videos on the seek bar R2. When a partial video is specified on the seek bar R2, the first image of the specified partial video is displayed in the partial video area R3, and normal and abnormal buttons are provided associated with each partial video area R3. When a partial video is specified, the user operates the normal or abnormal button for each specified partial video. When the save button is pressed on the screen shown in Figure 10B, the control unit 41 of the supervisor terminal 40 extracts the specified partial videos from the cycle video, stores the partial video where the normal button was pressed as the normal operation partial video, and the partial video where the abnormal button was pressed as the abnormal operation partial video, along with the cycle video, in the storage unit 42. The control unit 41 also sends the cycle video and each partial video to the server 50, and the server 50 performs a learning process based on the cycle video and each partial video acquired from the supervisor terminal 40. Here, the control unit 51 inputs the cycle video, the normal operation partial video, and the abnormal operation partial video into the learning model 12M, causes the learning model 12M to detect moving objects in the cycle video and partial videos, extracts image features for each unit time in the area where the moving object is captured, and adds information indicating normal or abnormal operation to the image features of the partial video and stores them.

[0047] The above-described process generates a learning model 12M that, when a video to be estimated is input, divides the input video into cycle videos of the same work as the normal cycle video and outputs information on the differences between each cycle video and the normal cycle video. The threshold and predetermined values ​​used in determining whether a location is a difference from the normal cycle video in the learning model 12M may be configurable. For example, if the control unit 51 of the server 50 receives input of the threshold and predetermined values ​​via the input unit, it may be configured to set the values ​​used in determining whether a location is a difference from the normal cycle video based on the input settings. Next, the process of detecting the occurrence of abnormal operation in real time based on the video of the worker captured by the camera 20 and notifying the worker's supervisor will be described using the learning model 12M generated by the above-described process.

[0048] Figure 11 is a flowchart showing an example of the abnormal operation detection process. In Figure 11, the processing performed by the camera 20 is shown on the left, the processing performed by the abnormality detection device 10 is shown in the center, and the processing performed by the notification terminal 30 is shown on the right. When a worker performs work in the workspace, they attach the camera 20 assigned to them to their body and turn on the power of the camera 20. When the power is turned on, the camera 20 starts up and begins taking pictures with its imaging unit (S41). If the camera 20 has a memory unit, it stores the captured images (videos) acquired by the imaging unit's shooting process in the memory unit. The camera 20 also transmits the captured images sequentially to the abnormality detection device 10 (S42). The camera 20 transmits the captured video along with the camera ID or the work ID of the worker using the camera 20 to the abnormality detection device 10.

[0049] The control unit 11 of the anomaly detection device 10 acquires the captured video transmitted sequentially from the camera 20 (S44), assigns a file name to it, and stores it in the storage unit 12. The control unit 11 uses the learning model 12M to divide the acquired captured video into cycle videos of the same operation as the normal cycle video (S45), and performs an anomaly detection process on each divided cycle video to detect the differences from the normal cycle video (S46). Here, the control unit 11 inputs the acquired captured video into the learning model 12M, acquires output information (detection results) from the learning model 12M, divides the captured video into cycle videos based on the acquired output information, and identifies the location of abnormal operation (difference from the normal cycle video) for each cycle video. The control unit 11 detects abnormal operation locations, including differences from a normal cycle video (locations that differ from normal operation, locations that resemble abnormal operation), cycle videos where the cycle time is above or below a predetermined value compared to the cycle time of a normal cycle video (work time of normal operation), locations where the worker's operation time is above or below a predetermined value compared to the operation time of normal operation, missing operations in the normal cycle video, and operations in the normal cycle video where the order of operations is reversed. The control unit 11 stores video information indicating the result of the abnormality detection process in the work line DB 12a, associated with the worker ID and camera ID (S47). Since the camera 20 transmits the captured video to the abnormality detection device 10 in real time while capturing it, the control unit 11 stores the start date and time of the captured video, the file name of the video data stored in the storage unit 12, the cycle time (start time and end time) of the cycle video split from the captured video, and the result of detecting abnormal operation for the cycle video (type of abnormal operation).

[0050] The control unit 11 determines whether the detection results stored in the work line DB 12a include abnormal operation (differences from the normal cycle video) (S48). If it determines that abnormal operation is included (S48: YES), it outputs an abnormality detection signal to the supervisor's notification terminal 30 (S49). For example, the control unit 11 reads the destination information of the notification terminal 30 carried by the supervisor from the work line DB 12a and uses the read destination information to output an abnormality detection signal to the supervisor's notification terminal 30.

[0051] When the notification terminal 30 receives an abnormality detection signal, it outputs an alert through notification processing by the notification unit (S50). For example, the notification terminal 30 outputs an alert by lighting or flashing a lamp, sounding a buzzer, or vibrating with a vibrator. The abnormality detection device 10 may also include the worker ID of the worker whose abnormal operation was detected in the abnormality detection signal, and if the abnormality detection signal includes the worker ID, the notification terminal 30 may output an alert by displaying the worker ID or the worker's name corresponding to the worker ID on the display unit or by outputting it as sound through the speaker. In this case, the supervisor can identify the worker who performed the abnormal operation as soon as the alert is notified, and can take appropriate action such as providing feedback to the worker at an early stage. Furthermore, if the abnormality detection device 10 has identified the type of abnormal operation that occurred in the abnormality detection processing in step S46, the control unit 11 may send an abnormality detection signal to the notification terminal 30 according to the identified type of abnormal operation. In this case, the notification terminal 30 can notify the type of abnormal operation that occurred by performing notification processing according to the received abnormality detection signal.

[0052] If the control unit 11 of the anomaly detection device 10 determines that the detection result does not include any abnormal operation (S48: NO), it skips step S49 and terminates the process. The camera 20 also determines whether to terminate the process, for example, when the power is turned off (S43). If it determines not to terminate the process (S43: NO), it returns to step S42 and transmits the images (recorded videos) captured by the shooting unit to the anomaly detection device 10 sequentially. If the camera 20 determines to terminate the process (S43: YES), it terminates the shooting process and the transmission of the recorded videos. When the camera 20 terminates the transmission of the recorded videos, it transmits the shooting termination date and time of the recorded videos to the anomaly detection device 10, and the control unit 11 of the anomaly detection device 10 stores the acquired shooting termination date and time in the video information of the work line DB 12a.

[0053] As described above, the video footage captured by the camera 20 is sequentially transmitted to the anomaly detection device 10, which can detect in real time whether the worker being filmed is performing an abnormal operation based on the video footage. Furthermore, if the anomaly detection device 10 detects an abnormal operation, it can notify the worker's supervisor, allowing the supervisor to grasp the occurrence of the abnormal operation early. Upon being notified of the occurrence of an abnormal operation, the supervisor uses the supervisor terminal 40 to check the video that has been determined to be an abnormal operation and decide whether the operation in the video is truly abnormal. If the supervisor determines that it is truly an abnormal operation, they use the video as a cycle video including the abnormal operation portion for training (retraining) the learning model 12M. If the supervisor determines that it is not an abnormal operation, they use the video as a cycle video including the normal operation portion for training (retraining) the learning model 12M.

[0054] Next, we will explain the process by which, when a supervisor is notified of an abnormal operation via the notification terminal 30, the supervisor terminal 40 checks the video that has been determined to be an abnormal operation and determines whether or not it is truly an abnormal operation. Figure 12 is a flowchart showing an example of the abnormal operation confirmation process procedure, and Figures 13 and 14 are explanatory diagrams showing example screens related to the confirmation process. In Figure 12, the processes performed by the abnormality detection device 10 are shown on the left, the processes performed by the supervisor terminal 40 are shown in the center, and the processes performed by the server 50 are shown on the right. When the abnormality detection device 10 detects the occurrence of an abnormal operation by any worker based on the video footage sequentially acquired from the camera 20, it executes the following process. For example, if the abnormality detection device 10 determines in step S48 of the process in Figure 11 that the detection result stored in the work line DB 12a includes an abnormal operation (S48: YES), it executes the following process, thereby performing the confirmation process by the supervisor.

[0055] The control unit 11 of the anomaly detection device 10 generates an anomaly list as shown in Figure 13 (S61). For example, the control unit 11 identifies video information for which the detection result is not normal from the video information of each worker stored in the work line DB 12a. The control unit 11 then obtains the worker ID, work space ID, work content (line information), date and time of the recorded video, file name, detection result, etc., corresponding to the identified video information from the work line DB 12a. Based on the obtained information, the control unit 11 generates an anomaly list as shown in Figure 13. The control unit 11 reads the video data of the recorded video from the storage unit 12 based on the file name of the recorded video, generates a thumbnail image using the first image (still image) of the read video data, and generates an anomaly list that displays the thumbnail image along with the file name.

[0056] The control unit 11 transmits the generated anomaly list to the supervisor terminal 40 (S62). The control unit 11 may transmit the anomaly list to the supervisor terminal 40 of the supervisor corresponding to the notification terminal 30 that output an anomaly detection signal in step S49 of the processing in Figure 11, or it may transmit the anomaly list to all supervisor terminals 40. The control unit 11 may also transmit the anomaly list to the supervisor terminal 40 in response to a request from the supervisor terminal 40. The control unit 41 of the supervisor terminal 40 acquires the anomaly list from the anomaly detection device 10 and displays the acquired anomaly list on the display unit 45 (S63). The control unit 41 stores the acquired anomaly list in the storage unit 42.

[0057] The anomaly list is configured to play the selected video when any of the recorded videos are selected, and the control unit 41 of the supervisor terminal 40 determines whether or not it has received a selection of any of the recorded videos via the anomaly list (S64). If the control unit 41 determines that it has not received a selection of any of the recorded videos (S64: NO), it waits until it does. If the control unit 41 determines that it has received a selection of any of the recorded videos (S64: YES), it generates a confirmation screen that displays the selected video (S65) and displays the generated confirmation screen on the display unit 45 (S66). Figure 14A shows an example of the confirmation screen, which displays the worker ID, worker name, workspace ID, workspace information, work details, shooting date and time (e.g., shooting start date and time), anomaly detection result (type of anomaly), the video in which the anomaly was detected, and a seek bar, etc. The confirmation screen also displays the image (partial video) in which the anomaly was detected, and the playback position of this partial image is indicated on the seek bar. The information displayed on the confirmation screen may be obtained by the control unit 41 from the abnormality detection device 10 (work line DB 12a) or from the server 50 (factory DB 52a). Furthermore, as shown in Figure 14A, the control unit 41 adds a mark M1 to one image (for example, the first still image) of the partial video in which abnormal operation was detected, at the location where the abnormal operation was detected (the location where the abnormal operation was detected), and displays it. Note that the location of the abnormal operation may be detected by the learning model 12M when it detects the partial video of the abnormal operation, or it may be detected by the control unit 11, for example, using the result of the learning model 12M detecting the partial video of the abnormal operation.

[0058] The supervisor checks the recorded video via the confirmation screen and determines whether the detected abnormal operation is truly abnormal. If it is determined not to be abnormal, the supervisor operates the "Save as Normal" button; if it is determined to be abnormal, the supervisor operates the "Save as Abnormal" button. Therefore, the control unit 41 determines whether the "Save as Normal" button or the "Save as Abnormal" button (collectively referred to as "buttons" in Figure 12) has been operated (S67). If it determines that neither button has been operated (S67: NO), it waits until either button is operated. If multiple locations identified as abnormal operations are displayed on the confirmation screen, the supervisor can, for example, operate the seek bar to display partial videos of other abnormal operations on the confirmation screen, confirm whether they are truly abnormal, and operate one of the buttons according to the confirmation result.

[0059] If the control unit 41 determines that any button has been operated (S67: YES), it generates a learning cycle video based on the currently displayed recorded video (S68). Here, if the "Save as Normal" button is operated, the control unit 41 treats the currently displayed portion of the video (the portion determined to be abnormal operation) as a portion of normal operation and generates a cycle video including this portion. Also, if the "Save as Abnormal" button is operated, the control unit 41 treats the currently displayed portion of the video as a portion of abnormal operation and generates a cycle video including this portion. The control unit 41 transmits the generated learning cycle video to the server 50 (S69), and the control unit 51 of the server 50 acquires the learning cycle video transmitted by the supervisor terminal 40 and stores it in the storage unit 52 (S70). The learning cycle video may be transmitted directly from the supervisor terminal 40 to the server 50, or it may be transmitted to the server 50 via the abnormality detection device 10. Furthermore, if the learning model 12M is being trained on a learning device other than the server 50, the supervisor terminal 40 transmits the learning cycle video to that learning device. The server 50 stores the cycle videos for training in the memory unit 52 each time it acquires them, and reads them from the memory unit 52 and performs the training process when retraining the learning model 12M.

[0060] Figure 14B shows another example of the confirmation screen. In the screen of Figure 14B, the recorded video selected via the abnormality list, a mark M2 indicating the start and end times of the cycle video included in the recorded video on the seek bar of the recorded video, and a mark M3 indicating the start and end times of the portion (partial video) in the recorded video that was judged to be abnormal are displayed on the seek bar. In the screen of Figure 14B, the supervisor can select one of the partial videos indicated by the mark M3 via the seek bar, and the selected partial video will be displayed. The supervisor checks the partial video displayed on the confirmation screen, determines whether the operation of the partial video is truly abnormal, and operates the "Save as Normal" button or the "Save as Abnormal" button according to the determination result. Even in this case, the control unit 41 of the supervisor terminal 40 generates a cycle video for training and sends it to the server 50 by performing steps S68 to S69 in Figure 12. Even in this case, the server 50 can acquire the cycle video for training and perform the training process of the training model 12M using the acquired cycle video for training.

[0061] Through the process described above, the supervisor can determine whether a partial video (cycle video) detected as abnormal by the anomaly detection device 10 is truly abnormal, and take appropriate action (e.g., providing feedback to the worker) based on the determination result. The supervisor can also set whether an operation is abnormal or normal, and this setting can be used in the learning process of the learning model 12M. By performing the learning process of the learning model 12M using partial videos (cycle videos) that have been set (annotated) as abnormal or normal by the supervisor, the accuracy of the learning model 12M can be improved. In the process described above, the anomaly detection device 10 generates an anomaly list related to the abnormal operation it detected and sends it to the supervisor terminal 40, but the anomaly list may also be generated by the server 50. Since the anomaly detection device 10 sends the anomaly detection result to the server 50, the server 50 can generate an anomaly list based on the anomaly detection result obtained from the anomaly detection device 10.

[0062] In the information processing system of this embodiment, the abnormal operation detection results from the abnormality detection device 10 are transmitted to the server 50 and stored in the factory DB 52a. Therefore, the server 50 can present the results of the work status, such as the rate of abnormal operation and the rate of normal operation, as the work results of each worker to, for example, a supervisor. In addition to supervisors, the recipients of the information may also be the person in charge of the work site (factory or branch) (for example, the factory manager), the manager in charge of each work site (for example, the person in charge at the head office), the personnel manager, etc.

[0063] Next, we will explain the process by which the server 50 presents the work results of each worker. Figure 15 is a flowchart showing an example of the work result presentation process procedure, and Figure 16 is an explanatory diagram showing an example of a screen related to the work result presentation process. In Figure 15, the processes performed by the server 50 are shown on the left, and the processes performed by the supervisor terminal 40 are shown on the right. For example, if a supervisor wants to check the work status of the workers they supervise, they access the server 50 using the supervisor terminal 40 and obtain the work results of each worker.

[0064] The control unit 41 of the supervisor terminal 40 receives the supervisor's designation of a worker whose work results they wish to check via the input unit 44 (S81), sends the worker ID of the designated worker to the server 50 (S82), and requests the work results of that worker. In addition to the worker whose work status the supervisor wishes to check, the control unit 41 may also obtain the period for which the work status of the worker to be checked, the workspace for which the work status of the worker to be checked, the work content for which the work status of the worker to be checked, etc., via the input unit 44. In this case, the control unit 41 sends the worker ID along with the information to be checked (information such as the period, workspace, and work content) to the server 50.

[0065] When the control unit 51 of the server 50 receives a request for work results from the supervisor terminal 40, it reads the total number of cycles and abnormal cycle information for the worker with the worker ID obtained from the supervisor terminal 40 from the factory DB 52a (S83). Here, when the control unit 41 obtains the information to be checked from the supervisor terminal 40, it reads the total number of cycles and abnormal cycle information corresponding to the obtained information. Based on the read total number of cycles and abnormal cycle information, the control unit 51 generates a work result screen as shown in Figure 16 (S84). The screen in Figure 16 displays, for each worker, the worker ID, worker name, workspace ID, workspace information, date, work details, total number of cycles, number of abnormal cycles and their percentage, number of normal cycles and their percentage, normal points, etc. Here, based on the abnormal cycle information read from the factory DB 52a, the control unit 51 counts the number of abnormal cycles including abnormal operations, and calculates the occurrence rate of abnormal cycles based on the counted number of abnormal cycles and the total number of cycles. The control unit 51 also calculates the number of normal cycles that do not include abnormal operations by subtracting the number of abnormal cycles from the total number of cycles, and calculates the normal cycle occurrence rate (normal score) based on the calculated normal cycle occurrence rate and the total number of cycles. Furthermore, the control unit 51 calculates normal points, which indicate how well the operator is performing normal operations, based on the normal cycle occurrence rate corresponding to each date. For example, the control unit 51 may calculate the average value of the normal cycle occurrence rate for each date and use that as the normal points. Alternatively, the control unit 51 may weight each date and use the weighted average as the normal points.

[0066] The control unit 51 transmits the generated work result screen to the supervisor terminal 40 (S85), and the control unit 41 of the supervisor terminal 40 acquires the work result screen transmitted by the server 50 and displays it on the display unit 45 (S86). This allows supervisors to provide the work results of workers whose work they wish to check. Through the above process, not only can feedback be provided early if each worker performs an abnormal operation, but the degree to which each worker performed a normal operation can be indicated by normal points. Therefore, personnel evaluations for each worker can be performed not only based on the rate of abnormal operation, but also on the rate of normal operation and normal points.

[0067] Conventionally, in factory and other workplaces, a method of monitoring workers' work status has been to use fixed cameras to film their work. However, when using fixed cameras, blind spots may occur depending on the positional relationship between the camera and the worker. In contrast, this embodiment uses a wearable camera to film the worker's hands and workspace, thereby obtaining high-precision video. In particular, when a wearable camera is attached to the worker's head (face), it is possible to film the work status as seen by the worker, resulting in video with even fewer blind spots. In this embodiment, the system uses video with suppressed blind spots to determine whether the worker's work status and movements are appropriate (normal or abnormal), thus obtaining more accurate judgment results.

[0068] The embodiment described above processes video footage of the worker's hands and workspace, but the processing target is not limited to the worker. For example, the system may be configured to detect abnormal movements (differences from a normal cycle video) in the work performed by a robot and the actions included in that work, based on video footage of the robot's workspace configured to perform a predetermined task. In this case, it is possible to monitor in real time whether the robot is performing the work and actions as configured.

[0069] The matters described in the embodiments described above can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. Moreover, although the claims use a form in which claims referencing two or more other claims (multi-claim form), it is not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used.

[0070] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended. [Explanation of symbols]

[0071] 10 Anomaly detection device 11 Control Unit 12 Storage section 13 Communications Department 17 Wireless Communication Section 20 cameras 30 Notification terminals 40 Supervisor terminal 41 Control Unit 42 Storage section 43 Communications Department 50 servers 51 Control Unit 52 Storage section 53 Communications Department 12M Learning Model

Claims

1. A video of the work area where the monitored object is working is captured by a camera attached to the monitored object, and this video is acquired for each of the monitored objects during their work. When a video of the workspace in which the target performs an action is input to a trained model that has been trained to detect whether or not there is abnormal operation in the action performed by the target, each of the acquired videos of the monitored target is input, and the detection result of whether or not there is abnormal operation is obtained for each monitored target. If abnormal operation is detected, an alert will be sent to the terminal of the administrator managing the monitored target. A program that instructs a computer to perform a process.

2. The system accepts the request for a normal video taken of the work space while the subject is performing normal work. Based on the specified normal video, when a video of the workspace where the subject performs work is input, the learning model is trained to detect the difference between the input video and the normal video. The program according to claim 1, which causes the computer to perform the processing.

3. The system accepts requests to specify the time periods during which the subject is operating normally and / or operating abnormally, based on a video recording of the workspace in which the subject performs its work. Based on the video which includes specified periods of normal operation and / or abnormal operation, when a video of the workspace where the subject performs work is input, the learning model is trained to detect the difference between the input video and the normal video. The program according to claim 2, which causes the computer to perform the processing.

4. On the seek bar indicating the playback position of the video of the workspace where the subject performs the work, the playback position of the difference between the video and the normal video is displayed. The system accepts a designation of normal operation or abnormal operation for the difference location displayed on the seek bar. Based on the video containing the difference portion where normal or abnormal operation is specified, when a video of the workspace where the target performs an operation is input, the learning model is trained to detect the difference between the input video and the normal video. The program according to claim 2 or 3, which causes the computer to perform the processing.

5. Based on the detection results for the presence or absence of abnormal operation, the image of the location where the abnormal operation was detected in the monitored video and the type of abnormal operation are output to the administrator's terminal. A program according to any one of claims 1 to 3 that causes the computer to perform processing.

6. From the monitored video, images of the locations where abnormal operation is detected are extracted. The extracted images are marked to indicate abnormal operation. The image with the aforementioned mark added, and the type of abnormal operation, are output to the administrator's terminal. The program according to claim 5, which causes the computer to perform the processing.

7. From the administrator's terminal, the system accepts a designation of normal operation or abnormal operation for the image of the location where abnormal operation is detected in the monitored video. Based on the video which includes the location where abnormal operation is detected, which is designated as either normal or abnormal operation, when a video of the workspace in which the subject performs an operation is input, the learning model is trained to detect whether or not there is abnormal operation in the operation performed by the subject in the input video. The program according to claim 5, which causes the computer to perform the processing.

8. For each of the monitored videos, a normal score is identified based on the detection results of whether or not abnormal operation is present. For each of the monitored targets, a normal score is calculated based on the normal score identified for the monitored video. A program according to any one of claims 1 to 3 that causes the computer to perform processing.

9. The learning model has been trained to detect as abnormal movements any part of the input video that differs from the part that has been trained as a normal movement, any part that is similar to the part that has been trained as an abnormal movement, any part where the difference between the operation time in the part that has been trained as a normal movement or the operation time in the normal video that has been trained as a normal operation is above or below a threshold, any part where there is a missing movement in the normal video that has been trained as a normal operation, or any part where the order of multiple movements in the normal video that has been trained as a normal operation is different. The program according to any one of claims 1 to 3.

10. The learning model accepts input of a threshold value for determining whether the input video is a difference from the normal video. Based on the input threshold, the learning model sets a threshold for determining the difference between the input video and the normal video. The program according to claim 2 or 3, which causes the computer to perform the processing.

11. The aforementioned monitored video includes multiple cycle videos taken while the monitored object performs a series of tasks. The learning model is trained to divide the input monitored video into cycle videos and to detect the presence or absence of abnormal operation for each cycle video. The program according to any one of claims 1 to 3.

12. The system accepts designations of normal video for each cycle video that is divided from the monitored video, or accepts designations of normal operation or abnormal operation for partial videos included in each cycle video. Based on the specified normal video, or based on the cycle video which includes the specified partial video which shows normal or abnormal operation, the learning model is trained to detect the difference between the input video and the normal video when a video of the workspace in which the target performs work is input. The program according to claim 11, which causes the computer to perform the processing.

13. A video of the work area where the monitored object is working is captured by a camera attached to the monitored object, and this video is acquired for each of the monitored objects during their work. When a video of the workspace in which the target performs an action is input to a trained model that has been trained to detect whether or not there is abnormal operation in the action performed by the target, each of the acquired videos of the monitored target is input, and the detection result of whether or not there is abnormal operation is obtained for each monitored target. If abnormal operation is detected, an alert will be sent to the terminal of the administrator managing the monitored target. An information processing method in which a computer performs the processing.

14. In an information processing device having a control unit, The control unit, A video of the work area where the monitored object is working is captured by a camera attached to the monitored object, and this video is acquired for each of the monitored objects during their work. When a video of the workspace in which the target performs an action is input to a trained model that has been trained to detect whether or not there is abnormal operation in the action performed by the target, each of the acquired videos of the monitored target is input, and the detection result of whether or not there is abnormal operation is obtained for each monitored target. If abnormal operation is detected, an alert will be sent to the terminal of the administrator managing the monitored target. Information processing device.

Images