Selection system, selection device, and selection method

Abstract

The present disclosure provides a selection system, a selection device, and a selection method capable of selecting an appropriate notification destination. A selection system according to the present disclosure comprises a detection unit, an acquisition unit, and a selection unit. The detection unit of the selection system detects an abnormality in work performed by a worker. The acquisition unit of the selection system acquires a work status of the work in which the abnormality has been detected. The selection unit of the selection system selects a notification destination to which the result of detecting the abnormality by the detection unit is notified on the basis of the work status acquired by the acquisition unit.

Description

Selection system, selection device, and selection method

[0001] The present disclosure relates to a selection system, a selection device, and a selection method.

[0002] There are known techniques for monitoring the work of workers. For example, Patent Literature 1 describes a technique for determining whether the work of a worker is correct based on an image of the work of the worker, and notifying the worker of the result of the determination.

[0003] Japanese Patent Application Laid-Open No. 2015-099022

[0004] However, in Patent Document 1, the notification destination for notifying the determination result of an abnormality in work, etc., is fixed, and the notification destination cannot be changed flexibly. For this reason, in Patent Document 1, it is difficult to notify the appropriate notification destination of the abnormality in work.

[0005] In view of the above-described problems, one of the objects of the present disclosure is to provide a selection system, a selection device, and a selection method that are capable of selecting an appropriate notification destination.

[0006] A selection system according to one aspect of the present disclosure includes a detection means for detecting abnormalities in work performed by a worker, an acquisition means for acquiring the work status of the work, and a selection means for selecting a notification destination to notify of the abnormality detection result based on the acquired work status.

[0007] A selection device according to one aspect of the present disclosure includes a detection means for detecting abnormalities in work performed by a worker, an acquisition means for acquiring the work status of the work, and a selection means for selecting a notification destination to notify of the detection result of the abnormality based on the acquired work status.

[0008] A selection method according to one aspect of the present disclosure detects an abnormality in work performed by a worker, acquires the work status of the work, and selects a notification destination to notify of the abnormality detection result based on the acquired work status.

[0009] According to the present disclosure, it is possible to select an appropriate notification destination.

[0010] 1 is a configuration diagram showing an example of a configuration of a selection system according to an embodiment. FIG. 1 is a configuration diagram showing an example of a configuration of a selection device according to an embodiment. FIG. 2 is a flowchart showing an example of a selection method according to an embodiment. FIG. 2 is a configuration diagram showing an example of a configuration of a monitoring system according to an embodiment. FIG. 3 is a configuration diagram showing an example of a configuration of an anomaly detection device according to an embodiment. FIG. 4 is a configuration diagram showing an example of a configuration of an anomaly detection unit according to an embodiment. FIG. 5 is a flowchart showing an example of operation of the anomaly detection device according to an embodiment. FIG. 6 is a flowchart showing a specific example of a notification destination selection process according to an embodiment. FIG. 7 is a diagram showing an example of a display on a management display according to an embodiment. FIG. 8 is a diagram showing another example of a display on a management display according to an embodiment. FIG. 9 is a diagram showing another example of a display on a work display according to an embodiment. FIG. 10 is a diagram showing another example of a display on a management display according to an embodiment. FIG. 11 is a configuration diagram showing an example of a configuration of an anomaly detection device according to an embodiment. FIG. 12 is a flowchart showing an example of operation of the anomaly detection device according to an embodiment. FIG. 13 is a diagram showing an example of a table used in a notification destination selection process according to an embodiment. FIG. 14 is a configuration diagram showing an example of a configuration of an anomaly detection device according to an embodiment. FIG. 15 is a configuration diagram showing an example of a configuration of hardware of a computer according to an embodiment.

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant description will be omitted as necessary.

[0012] (Overview of the Embodiments) First, an overview of the embodiments will be described. Fig. 1 shows an example configuration of a selection system 10 according to some embodiments. The selection system 10 can be applied to, for example, a system that notifies of abnormalities in work when monitoring the work of workers on a production line.

[0013] The selection system 10 is a means for selecting a notification destination of an abnormality in a work operation. In the example of FIG. 1 , the selection system 10 includes a detection unit 11, an acquisition unit 12, and a selection unit 13. The detection unit 11 detects an abnormality in the work performed by a worker. For example, the detection unit 11 may analyze an image captured by a camera installed on the production line to detect an abnormality in the work. The detection unit 11 may also detect an abnormality in the work operation based on measurement results from a sensor that measures the state of a work object including a part to be worked on, a work tool used in the work, and the like.

[0014] The acquisition unit 12 acquires the work status of the work in which the detection unit 11 detects an abnormality. The work status is a status related to the progress of the work or the worker performing the work. For example, the work status may be a progress status indicating the progress of the work. The work progress status indicates the progress of the work on the work object in which the abnormality was detected. For example, the work progress status indicates the work process and worker performing the work on the work object. The work status may also include a status indicating the attributes of the worker. The worker status indicates the worker's level for the work. For example, the worker status may be work experience, years of service, proficiency, error frequency, etc. The worker status may be an indicator to distinguish between a new employee and a veteran, etc.

[0015] The selection unit 13 selects a notification destination to notify of the result of the abnormality detected by the detection unit 11, based on the work status acquired by the acquisition unit 12. For example, when the work status includes the progress state of the work, the selection unit 13 may select, as the notification destination, a worker who is working on the work object of the work in which the abnormality was detected. The selection unit 13 may select, depending on the progress state of the work, a worker in the work process in which the abnormality was detected, or a worker in the next work process or later, as the notification destination. Furthermore, when the work status includes the status of the worker, the selection unit 13 may select, depending on the status of the worker, a notification destination from among the worker in the work in which the abnormality was detected, an instructor who is instructing the worker, and a manager who is managing the work.

[0016] The selection system 10 may be configured with one device or multiple devices. Fig. 2 shows an example configuration of a selection device 20 according to some embodiments. In the example of Fig. 2, the selection device 20 includes the detection unit 11, acquisition unit 12, and selection unit 13 shown in Fig. 1. For example, part or all of the selection system 10 or the selection device 20 may be arranged in an edge device arranged at an edge or in a cloud server arranged in the cloud.

[0017] 3 shows an example of a selection method according to some embodiments. For example, the selection method according to some embodiments may be performed by the selection system 10 of FIG. 1 or the selection device 20 of FIG.

[0018] 3, the detection unit 11 detects an abnormality in the work performed by a worker (S11). For example, the detection unit 11 detects an abnormality in the work based on an image captured by a camera installed on the production line or a measurement result from a sensor that measures the state of the work object, work tools, etc.

[0019] Next, the acquisition unit 12 acquires the work status of the work (S12). For example, the acquisition unit 12 acquires the progress state of the work, the status of the worker, etc. as the work status.

[0020] Next, the selection unit 13 selects notification recipients to notify of the abnormality detection result based on the acquired work status (S13). For example, the selection unit 13 selects notification recipients from among the workers in the work process or workers in the next work process or later, depending on the progress of the work. Furthermore, the selection unit 13 selects notification recipients from among the workers performing the work, the instructor, and the manager, depending on the status of the workers.

[0021] When an abnormality occurs during work, if the notification of the abnormality is not given appropriately, the worker may not notice the abnormality. Also, if the notification of the abnormality is given to workers who are not involved in the detected abnormality or if there are too many notifications of the abnormality, the worker may intentionally ignore the notification of the abnormality.

[0022] Therefore, in the embodiment, as described above, when an abnormality is detected, the notification destination of the abnormality is selected according to the work situation, such as the progress of the work and the status of the worker. This makes it possible to notify the abnormality to an appropriate notification destination. By notifying the abnormality to an appropriate notification destination, the notification destination can reliably grasp the abnormality that the notification destination needs.

[0023] (First Embodiment) Next, a description will be given of a first embodiment. This embodiment is an example in which a notification destination of an abnormality is selected based on the progress of the work.

[0024] FIG. 4 shows an example configuration of a monitoring system 1 according to some embodiments. The monitoring system 1 is, for example, a system that monitors the work of workers using video captured by cameras installed on a production line. The production line is a line that manufactures target products through multiple work processes. For example, the target products may be, but are not limited to, home appliances such as televisions, electronic devices such as personal computers, electronic components such as semiconductor devices, mobile devices such as automobiles, processed foods, etc. Note that video includes multiple images (frames) in a time series, so the terms video and image are interchangeable. In other words, the monitoring system 1 can be considered both as a video processing system that processes video captured by cameras and as an image processing system that processes images captured by cameras.

[0025] In the example of FIG. 4 , the monitoring system 1 includes an anomaly detection device 100 , a work display 210 , a camera 220 , a sensor 230 , and a management display 310 .

[0026] A work display 210, a camera 220, and a sensor 230 are arranged at each work process in the production line. A work process includes a work area where a worker performs work. It can be said that a work display 210, a camera 220, and a sensor 230 are arranged for each worker. In addition, a management display 310 is arranged at a location where a manager manages (monitors) the work. The location where the work is managed may be the same manufacturing site as the production line including each work process, or may be a location away from the manufacturing site. When multiple managers manage the work, multiple management displays 310 may be arranged.

[0027] The anomaly detection device 100 may be located in the same manufacturing site as the manufacturing line including each work process, or may be located at a location remote from the manufacturing site. Furthermore, the anomaly detection device 100 may be located at the same location as the management display 310, or may be located at a location remote from the management display 310.

[0028] The anomaly detection device 100 is communicably connected to the work display 210, the camera 220, the sensor 230, and the management display 310 via any network. The network may be a wired network, a wireless network, or a combination thereof. Examples of wired networks include the Internet, a wide area network (WAN), and a LAN. Examples of wireless networks include 4G, local 5G / 5G, long term evolution (LTE), and a wireless LAN. The work display 210, the camera 220, and the sensor 230 are communicably connected to the management display 310 via the network.

[0029] The camera 220 captures images of the work area of ​​each work process. For example, the camera 220 can capture images of a workbench (work table) from above, capturing images of the worker working, the worker's hands, the work object, the work tools, etc. The camera 220 may be a fixed camera secured to a predetermined position, a wearable camera worn by a worker, or a camera equipped on a robot having an autonomous mobile mechanism. The camera 220 may capture images of the work area of ​​one work process, or may capture images of the work areas of multiple work processes. The camera 220 transmits the captured images including the work areas to the anomaly detection device 100 via a network. The camera 220 may transmit the captured images to the work display 210 or the management display 310.

[0030] The sensors 230 are various sensors that measure (measure) the conditions of the work object, work tools, work equipment, etc. in each work process. The sensors 230 may measure the temperature, vibration, pressure, etc. of the work object, work tools, work equipment, etc. The sensors 230 may be sensors arranged for measurement or may be measuring instruments attached to the work tools. The sensors 230 transmit sensor data including the measured values ​​to the anomaly detection device 100 via a network.

[0031] The anomaly detection device 100 detects an abnormality in a work operation based on information collected from each work process and notifies the user of the detected abnormality. The anomaly detection device 100 may be one or more physical servers, or may be a virtualized server built on a virtualization platform. The anomaly detection device 100 may be a cloud server on the cloud or an on-premise server installed at the work site. The anomaly detection device 100 may receive video of the work area captured by the camera 220 at each work process and detect an abnormality in the work operation by analyzing the received video. The anomaly detection device 100 may also receive sensor data measured by the sensor 230 at each work process and detect an abnormality in the work operation based on the received sensor data. The anomaly detection device 100 may also determine the progress of the work operation by analyzing the received video.

[0032] The anomaly detection device 100 also functions as a selection device that selects a notification destination for an anomaly. In this example, the anomaly detection device 100 selects a notification destination for notifying the anomaly detection result based on the progress state determination result. The anomaly detection device 100 may select a notification destination for the anomaly detection result based on the anomaly detection result and the progress state determination result. For example, the notification destination is a worker or a manager of each work process. Specifically, the notification destination is the work display 210 of each worker or the management display 310 of the manager, which is the destination of the anomaly detection result. The anomaly detection device 100 transmits the anomaly detection result to the selected notification destination display via a network.

[0033] The work display 210 is a work display device that displays information necessary for the worker to perform each work process. The work display 210 is also an output device that outputs work abnormalities to the worker. The work display 210 receives the detection result of the corresponding work abnormality from the abnormality detection device 100 and displays the received abnormality detection result on a GUI (Graphical User Interface), which is a display screen. The work display 210 may display an image of the work process of its own device as well as the abnormality detection result. The work display 210 may include an input unit that accepts input operations from the worker, a control unit that processes the received and input information and controls the information to be output, etc. The work display 210 may be a tablet terminal, a personal computer, etc.

[0034] The management display 310 is a management display device that displays information necessary for a manager to monitor the work of workers. The management display 310 is also an output device that outputs work abnormalities to the manager. The management display 310 receives abnormality detection results for multiple work processes from the abnormality detection device 100 and displays the received abnormality detection results on a GUI, which is a display screen. The management display 310 may display images of multiple work processes as well as abnormality detection results. The management display 310 may include an input unit that accepts input operations from the manager, a control unit that processes received and input information and controls output information, etc. The management display 310 may be a tablet terminal, a personal computer, etc.

[0035] The work display 210 and the management display 310 are examples of output devices that output (notify) abnormalities in work. The work display 210 and the management display 310 are not limited to devices that display abnormalities, but may also be devices that output abnormalities using a lamp light or sound such as a voice or warning sound.

[0036] 5 shows an example configuration of an anomaly detection device 100 according to some embodiments. Note that the configuration in FIG. 5 is just an example, and other configurations may be used as long as they are capable of executing the operation examples described below.

[0037] In the example of FIG. 5, the anomaly detection device 100 includes an image acquisition unit 110, a sensor data acquisition unit 120, an anomaly detection unit 130, a progress status determination unit 140, a notification destination selection unit 150, and a notification unit 160.

[0038] The video acquisition unit 110 acquires video captured by the camera 220 from the camera 220 at each work process via a network. The video captured by the camera 220 is also referred to as work video. For example, the work video includes video of a work area in one work process, but may also include video of work areas in multiple work processes. The video acquisition unit 110 also functions as an image acquisition unit that acquires multiple images, i.e., frames, in time series.

[0039] The sensor data acquisition unit 120 acquires sensor data including measurement values ​​measured by the sensors 230 from the sensors 230 in each work process via a network.

[0040] The anomaly detection unit 130 analyzes the work video acquired from the camera 220 and the sensor data acquired from the sensor 230 to detect an abnormality in the work of each work process. For example, the anomaly detection unit 130 may detect an abnormality in the work performed by a worker, an abnormality in the work object, or an abnormality in the work environment including work tools and work equipment. The anomaly detection unit 130 outputs the details of the detected abnormality, information on the work object, etc. as the detection result.

[0041] FIG. 6 shows an example configuration of the anomaly detection unit 130 according to some embodiments. In the example of FIG. 6, the anomaly detection unit 130 includes a worker determination unit 131, a work determination unit 132, and a work environment determination unit 133. The anomaly detection unit 130 may detect anomalies using a single analysis engine that includes the functions of the worker determination unit 131, the work determination unit 132, and the work environment determination unit 133. Note that the anomaly detection unit 130 may include only one of the worker determination unit 131, the work determination unit 132, and the work environment determination unit 133. Furthermore, the anomaly detection unit 130 may detect anomalies using methods other than those using the worker determination unit 131, the work determination unit 132, and the work environment determination unit 133.

[0042] The worker determination unit 131 determines abnormalities in the work performed by the worker. For example, the worker determination unit 131 may detect abnormalities in the work performed by the worker by analyzing a work video. The worker determination unit 131 may analyze the hand movements (hand movements) of the worker from the work video and detect abnormalities in the work based on the analysis results of the hand movements. The hand movement analysis may use an object recognition engine using machine learning such as deep learning. For example, the object recognition engine may recognize objects in an image and analyze the work of the worker based on the hand positions of the recognized person (worker). For example, the hand movement analysis may analyze the work status, including when the work is in operation or stopped, and the cycle time from the start to the end of the work. For example, the worker determination unit 131 may detect abnormalities based on the difference between the cycle time obtained by the hand movement analysis and a predetermined work time. The predetermined work time may be, for example, the time designed when the production line was designed.

[0043] The worker determination unit 131 may also analyze the worker's finger movements (finger movements) from the work video and detect an abnormality in the work based on the analysis results of the finger movements. The finger movement analysis may use a skeleton estimation engine, an object recognition engine, and a finger movement recognition engine that use machine learning such as deep learning. For example, the skeleton estimation engine may estimate the finger skeleton in the image, and the object recognition engine may recognize the work parts, work tools, etc. in the image. Furthermore, the finger movement recognition engine may recognize the work content, etc., based on the co-occurrence relationship between the feature amounts of the finger skeleton and the feature amounts of the work parts, work tools, etc. For example, the worker determination unit 131 may detect an abnormality based on whether the work content obtained by the finger movement analysis matches a predetermined work content. The worker determination unit 131 may detect an abnormality when a work process includes multiple tasks, when the order of the tasks differs from the predetermined order, or when there are fewer or more tasks than the predetermined tasks.

[0044] The worker determination unit 131 may also analyze the worker's behavior from the work video and detect an abnormality in the work based on the worker's behavior. The behavior analysis may use a posture estimation engine, an object recognition engine, and a behavior recognition engine that use machine learning such as deep learning. For example, the posture estimation engine estimates the posture of a person (worker) in an image, and the object recognition engine recognizes work parts, work tools, etc. in the image. Furthermore, the behavior recognition engine recognizes the person's behavior based on a co-occurrence relationship between the feature amount of the person's posture and the feature amount of the work parts, work tools, etc. For example, the worker determination unit 131 may detect an abnormality based on whether the person's behavior obtained by the behavior analysis matches a predetermined behavior. The worker determination unit 131 may detect an abnormality when a worker leaves their desk and cannot be detected from the image. The worker determination unit 131 may detect an abnormality when the number of workers performing a work process differs from a predetermined number.

[0045] The work object determination unit 132 determines an abnormality in a work object on which a worker is working. For example, the work object determination unit 132 may detect an abnormality in a work object by analyzing a work video. The work object includes a part on which a worker is working, a product being assembled, etc. For example, the work object may be recognized by an object recognition engine, and an abnormality in the work object may be detected depending on whether the shape of the recognized work object matches a specified shape.

[0046] The work object determination unit 132 may also detect an abnormality in the work object by analyzing sensor data. For example, the sensor 230 may measure the temperature, vibration, pressure, etc. of the work object, and detect an abnormality in the work object based on the difference between the measured value and a predetermined reference value.

[0047] Furthermore, the work environment determination unit 133 determines an abnormality in the work environment in which the worker is working. For example, the work environment determination unit 133 may detect an abnormality in the work environment by analyzing work video. The work environment includes work tools, work facilities, etc. For example, a work tool may be recognized by an object recognition engine, and an abnormality in the work tool may be detected depending on whether the shape of the recognized work tool matches a specified shape.

[0048] The work environment determination unit 133 may also detect abnormalities in the work environment by analyzing sensor data. For example, the sensor 230 may measure the temperature, vibration, pressure, etc. of the work tool or work equipment, and detect an abnormality in the work tool or work equipment based on the difference between the measured value and a predetermined reference value.

[0049] Returning to FIG. 5 , the progress status determination unit 140 analyzes the video acquired from the camera 220 and determines the progress status of work in each work process. For example, the progress status determination unit 140 analyzes the work video of each work process and detects the progress status of work on the work object in each work process. The progress status of work is the status of the work object on which work is being performed in each work process. For example, an object recognition engine recognizes the work object from the video and detects the work object being worked on in each work process. Note that the progress status of work may be determined by other methods other than video analysis. Information on the work object on which the worker is working in each work process may be input, and the progress status of work may be determined based on the input information, etc.

[0050] The notification destination selection unit 150 selects a notification destination to notify of the abnormality detection result based on the progress state determination result by the progress state determination unit 140. For example, the notification destination selection unit 150 selects a notification destination of the abnormality based on the work process (worker) performing the work on the work object of the work in which the abnormality was detected.

[0051] For example, if the work object of the work in which the abnormality was detected is currently being worked on in the same work process as the work process in which the abnormality was detected, the notification destination selection unit 150 may set the worker of the work in which the abnormality was detected as the notification destination. Furthermore, if an immediate notification of the detected abnormality is required, the worker of the work in which the abnormality was detected may be set as the notification destination. For example, if the detected abnormality is due to a minor mistake and can be corrected immediately, or if the work needs to be stopped urgently, the worker of the work in which the abnormality was detected may be set as the notification destination. In other words, the notification destination selection unit 150 may select a worker as the notification destination depending on the detected abnormality.

[0052] For example, if the work object of the work in which the abnormality was detected has progressed to the next work process or later and the worker at the source of the abnormality is unable to respond, the notification destination selection unit 150 may set the worker in the work process working on the work object as the notification destination. By selecting a worker in the next work process or later as the notification destination, it is possible to request the worker in the next work process or later to remove or correct the work object. Furthermore, if a long time passes between the occurrence of the abnormality and its detection, the notification destination may be the worker in the work process working on the work object in which the abnormality occurred. Furthermore, the notification destination may be the worker in the work process working at the current position of the work object estimated from the elapsed time since the work in which the abnormality was detected.

[0053] Furthermore, the notification destination selection unit 150 may select a manager as the notification destination, in addition to the worker performing the work in which the abnormality was detected or the worker in the next work process or later. For example, if the detected abnormality does not require an immediate response, the manager may be selected as the notification destination. For example, the manager may be selected as the notification destination when it is desired to collect information about a work object on which abnormal work has been performed. In other words, the notification destination selection unit 150 may select a manager as the notification destination depending on the detected abnormality.

[0054] The notification unit 160 notifies the selected notification destination of the abnormality detection result. For example, the abnormality detection result is transmitted via a network to the work display 210 of the selected notification destination worker or the management display 310 of the manager. The notification unit 160 may transmit the work video acquired from the camera 220 to the notification destination display.

[0055] 7 shows an example of the operation of the anomaly detection device 100 according to some embodiments. Note that the process flow in FIG. 7 is an example, and the order of some of the processes may be changed. For example, S101 and S102 may be executed in parallel, or one of them may be executed first.

[0056] 7 , the anomaly detection device 100 acquires video from the cameras 220 in each work process (S101). The cameras 220 in each work process capture images of the work area of ​​each work process and transmit the captured images to the anomaly detection device 100. The video acquisition unit 110 acquires the images transmitted from the multiple cameras 220, i.e., the work videos, via the network. For example, the work videos include images of the workers performing work in the work area, the workers' hands, the work objects, etc.

[0057] Furthermore, the anomaly detection device 100 acquires sensor data from the sensors 230 in each work process (S102). The sensors 230 in each work process perform various measurements in each work process and transmit sensor data including measurement values ​​to the anomaly detection device 100. The sensor data acquisition unit 120 acquires the sensor data transmitted from the multiple sensors 230 via a network. For example, the sensor data includes temperature, vibration, pressure, etc. of the work object, work tools, work equipment, etc.

[0058] Next, the anomaly detection device 100 detects an anomaly in the work of each work process (S103). For example, an anomaly is detected by the worker determination unit 131, the work object determination unit 132, and the work environment determination unit 133 in the anomaly detection unit 130. An anomaly may be detected by any one of the worker determination unit 131, the work object determination unit 132, and the work environment determination unit 133, or by combining the determination results of each determination unit.

[0059] For example, the worker determination unit 131 may perform hand movement analysis from the work video and detect an abnormality if the difference between the analyzed cycle time and a predetermined work time is greater than a predetermined value, if the difference between the average cycle time and the predetermined work time is greater than a predetermined value, or if the variance of the cycle time is greater than a predetermined value. The worker determination unit 131 may also perform finger movement analysis from the work video and detect an abnormality if the analyzed work content of the worker differs from the predetermined work content. The worker determination unit 131 may also perform behavior analysis from the work video and detect an abnormality if the analyzed behavior of the worker differs from the predetermined behavior.

[0060] The task determination unit 132 may also recognize a task object from the task video and detect an abnormality if the recognized task object shape does not match the specified shape. The task determination unit 132 may also detect an abnormality if the difference between the measurement value of the sensor data acquired from the sensor 230 and the reference value is greater than a predetermined value.

[0061] The work environment determination unit 133 may also recognize a work tool from the work video and detect an abnormality if the recognized shape of the work tool does not match the specified shape. The work environment determination unit 133 may also detect an abnormality if the difference between the measurement value of the work tool or work equipment acquired from the sensor 230 and the reference value is greater than a predetermined value.

[0062] The anomaly detection device 100 also determines the progress of work in each work process (S104). For example, the progress determination unit 140 recognizes work objects from the work video and detects work objects being worked on in each work process. The progress determination unit 140 determines the progress of the work object for which an anomaly has been detected.

[0063] Next, the anomaly detection device 100 selects a notification destination of the anomaly detection result (S105). The notification destination selection unit 150 selects a notification destination of the anomaly based on the progress state of the work in which the anomaly was detected.

[0064] FIG. 8 illustrates a specific example of the notification destination selection process (S105) according to some embodiments. In the example of FIG. 8 , the notification destination selection unit 150 determines the progress status of the work object for the work in which the abnormality was detected (S110). If the work object for the work in which the abnormality was detected is currently being worked on in the same work process, the notification destination selection unit 150 selects the worker in the work process in which the abnormality was detected as the notification destination (S111). Furthermore, if the work object for the work in which the abnormality was detected is being worked on in the next work process or later, the notification destination selection unit 150 selects the worker in the next work process or later as the notification destination (S112). In other words, the notification destination selection unit 150 selects the worker currently working on the work object for the work in which the abnormality was detected as the notification destination. Note that the notification destination selection unit 150 may select a manager in addition to the worker as the notification destination. The notification recipient selection unit 150 may select the worker and the manager as the notification recipients in either or both cases where the work object for the work in which the abnormality was detected is currently being worked on in the same work process, or where the worker in the next work process or later is selected as the notification recipient.

[0065] 7 , following the selection of the notification destination, the anomaly detection device 100 notifies the selected notification destination of the anomaly detection result (S106). If a worker in a work process is selected as the notification destination, the notification unit 160 transmits information about the detected anomaly to the work display 210 for the corresponding work process, and the work display 210 displays the received information about the anomaly. Furthermore, if a manager is selected as the notification destination, the notification unit 160 transmits information about the detected anomaly to the management display 310, and the management display 310 displays the received information about the anomaly. For example, the notification unit 160 may generate display information to be displayed on each display, as in the following display example, and output the generated display information to each display.

[0066] 9 shows an example of a display on the management display 310 according to some embodiments. In the example of FIG. 9, video of each work process is displayed on the display screen of the management display 310. For example, the management display 310 receives video from the camera 220 of each work process and displays the received video in a matrix. Furthermore, the management display 310 highlights video of a work process in which an abnormality has been detected, based on the abnormality detection result received from the anomaly detection device 100. In this example, the highlighting is achieved by displaying the video of the work process in which an abnormality has been detected in a bold colored frame (e.g., red) and displaying an icon indicating the abnormality.

[0067] FIG. 10 shows another example of the display of the management display 310 according to some embodiments. In the example of FIG. 10, images of each work process are displayed in a matrix on the display screen of the management display 310, and furthermore, the image of the work process in which an abnormality has been detected is highlighted. In this example, the highlighting is achieved by enlarging the image of the work process in which an abnormality has been detected and displaying it larger than the images of the other work processes. In this case, the display of the images of the other work processes may be omitted. Furthermore, as in FIG. 9, the image of the work process in which an abnormality has been detected is displayed with a colored bold frame, and an icon indicating the abnormality is also displayed.

[0068] FIG. 11 shows another example of a display on the management display 310 according to some embodiments. In the example of FIG. 11 , the display screen of the management display 310 displays a video of the work process in which an abnormality was detected, and further highlights the location of the abnormality. The management display 310 receives video from the camera 220 of the work process in which the abnormality was detected, displays the received video, and further highlights the location of the abnormality based on the abnormality detection result received from the anomaly detection device 100. For example, the highlighting is displayed by surrounding the location of the abnormality with a colored (e.g., red) circle. In addition, the display screen displays detailed information about the work process and the abnormality, a confirmed skip button, a notify person in charge button, and the like. When the manager confirms the displayed abnormality and clicks the confirmed skip button, the display of the abnormality ends. When the manager clicks the notify person in charge button, the abnormality is notified to the worker's work display 210. The processes required for the confirmed skip and the notify person in charge may be executed by the anomaly detection device 100 or the management display 310.

[0069] FIG. 12 shows an example of a display on the work display 210 according to some embodiments. In the example of FIG. 12, the work display 210 displays a video of the corresponding work process and further highlights the location of the abnormality. The work display 210 receives video from the camera 220 of the work process of the device itself, displays the received video in a row, and further highlights the location of the detected abnormality based on the abnormality detection result received from the anomaly detection device 100. For example, as shown in FIG. 11 , the highlighting is displayed by surrounding the location of the abnormality with a colored circle. In addition, the display screen displays detailed information about the work process and the abnormality, a confirmed skip button, a work restart button, and the like. When the worker confirms the displayed abnormality and clicks the confirmed skip button, the administrator's management display 310 is notified that the abnormality has been confirmed, and the display of the abnormality ends. The worker may also confirm the displayed abnormality, click the work restart button, and start the current work again from the beginning. The processing required to skip the confirmed abnormality or restart the work may be executed by the anomaly detection device 100 or the work display 210. As shown in Figures 11 and 12, the information notified to the manager and the information notified to the worker may be different. That is, the information to be notified may change depending on the notification destination. For example, when notifying the manager, the worker's name may be displayed as shown in Figure 11, and when notifying the worker of each process, the worker's name may not be displayed as shown in Figure 12. Furthermore, when notifying the worker who will correct the abnormality and other workers, the abnormality may be displayed to the other workers, and the correction procedure may be displayed to the worker who will correct the abnormality, in addition to the display of the abnormality.

[0070] FIG. 13 illustrates another example of a display on the management display 310 according to some embodiments. In the example of FIG. 13 , the management display 310 displays images of each work process corresponding to the flow of the production line on its display screen. For example, the management display 310 receives images from the cameras 220 of each work process and displays the received images in order of the flow of the production line. The management display 310 may acquire the cycle time of each work process from the anomaly detection device 100 and display the acquired cycle time in association with each work process. Furthermore, the management display 310 highlights the location of an abnormality in the image of the work process in which an abnormality was detected, based on the anomaly detection result received from the anomaly detection device 100. For example, as shown in FIG. 11 , the highlighting may be performed by surrounding the location of the abnormality with a colored circle. If the cycle time is abnormal, the corresponding cycle time may be highlighted. By displaying the production line in a bird's-eye view, the work can be evaluated throughout the entire assembly process.

[0071] In addition, when multiple abnormalities are detected on the management display 310 shown in FIGS. 9 to 11 and 13 , the priority of the information to be displayed and the information to be highlighted can be selected. The selection of the priority of the information to be displayed and the information to be highlighted may be performed by either the anomaly detection device 100 or the management display 310. As an example, when multiple abnormality notifications are received, the anomaly detection device 100 may select the target to be highlighted based on the analysis results of the work video. Specifically, areas or images with a high degree of abnormality may be selected preferentially based on the cycle time (work time) obtained by manual movement analysis. For example, a cycle waveform indicating the speed and speed of the work may be generated by manual movement analysis, and areas or images with abnormal cycle waveforms may be selected preferentially. Cycles can be detected from the periodicity of the cycle waveform, and the time per cycle can be measured. For example, if the cycle waveform is more irregular than a predetermined waveform, it may be determined that there is a problem with the work, and the corresponding area or image may be selected preferentially. Furthermore, when a task takes a long time based on the cycle time, when the task time is extremely short, or when the task time deviates from the average value, the corresponding area or image may be selected with priority. As another example, the target to be highlighted may be selected according to the status (attributes) of the worker. For example, abnormalities of workers with short years of service or low skill level may be highlighted with priority.

[0072] As described above, in this embodiment, an abnormality in work is detected from images captured by a camera, a notification destination of the abnormality is selected based on the progress of the work, and the abnormality is notified to the selected notification destination. This makes it possible to select a worker in a work process who can respond to the abnormality as the notification destination depending on the progress of the work, so that the abnormality can be notified to the appropriate notification destination.

[0073] (Embodiment 2) Next, embodiment 2 will be described. In this embodiment, an example is shown in which a notification destination of an abnormality is selected based on the status of the worker. Note that this embodiment can be implemented in combination with embodiment 1, and each configuration shown in embodiment 1 may be used as appropriate.

[0074] Fig. 14 shows an example configuration of an anomaly detection device 100 according to some embodiments. In the example of Fig. 14, the anomaly detection device 100 includes a worker information acquisition unit 170 instead of the progress state determination unit 140, as compared to the example of Fig. 5. The anomaly detection device 100 may also include the progress state determination unit 140 and the worker information acquisition unit 170. In this case, the notification destination selection unit 150 may select a notification destination for the anomaly detection result based on the progress state of the work and information about the worker.

[0075] The worker information acquisition unit 170 acquires information about workers in each work process. The worker information acquisition unit 170 may acquire the information about workers from a database within the anomaly detection device 100 or from an external system. The information about workers is information about the status (attributes) of the worker. The status may be work experience, years of service, proficiency, and error frequency. The status is not limited to numerical values ​​such as work experience and years of service, but may also be a work level (rank) such as rookie or veteran depending on work experience and years of service. In this example, the notification destination selection unit 150 selects a notification destination to notify of the anomaly detection result based on the acquired information about the worker.

[0076] Fig. 15 shows an example of the operation of an anomaly detection device 100 according to some embodiments. In the example of Fig. 15, similar to the example of Fig. 7, the anomaly detection device 100 acquires video from the camera 220 at each work process (S101), acquires sensor data from the sensor 230 at each work process (S102), and detects an anomaly in the work at each work process (S103).

[0077] The anomaly detection device 100 also acquires information about the worker in each work process (S201). For example, the worker information acquisition unit 170 acquires the status of the worker performing the work in which the anomaly was detected from a database or the like.

[0078] Next, the anomaly detection device 100 selects a notification destination for the anomaly detection result (S105). The notification destination selection unit 150 selects a notification destination for the anomaly based on the status of the worker performing the task in which the anomaly was detected. For example, the notification destination selection unit 150 may set a notification destination for each worker status in a table or the like in advance. Note that the example using a table is not limited to this, and the notification destination may also be selected using a learning model or the like that has been machine-learned to determine a notification destination according to the worker status.

[0079] Fig. 16 shows an example of a status table for setting notification destinations. In the example of Fig. 16, a notification destination is associated with each worker's status. For example, when an abnormality is detected in an operation, the notification destination selection unit 150 refers to the status table of Fig. 16 and determines a notification destination according to the status of the worker who performed the abnormal operation.

[0080] In this example, if the worker's status is new, the selected notification recipients are the new worker who performed the work, the veteran (instructor) who is instructing the new worker, and the manager. For example, if an abnormality occurs due to a new worker's mistake, notifying the new worker allows for educational guidance, notifying a designated veteran allows for follow-up and guidance on the new worker's work, and notifying the manager allows for data on the new worker's mistakes to be collected and used for future guidance. Also, in this example, if the worker's status is veteran, only the veteran who performed the work is selected as the notification recipient. For example, if an abnormality occurs due to a minor mistake, notifying only the worker in charge can reduce the burden on the manager.

[0081] 7 , the anomaly detection device 100 then notifies the selected notification destination of the anomaly detection result (S106). If a new employee is selected as the notification destination, the notification unit 160 transmits information about the detected anomaly to the work display 210 of the work process where the new employee is working. If a veteran is selected as the notification destination, the notification unit 160 transmits information about the detected anomaly to the work display 210 of the work process where the veteran is working.

[0082] As described above, in this embodiment, the notification destination of the abnormality is selected based on the status of the worker performing the work in which the abnormality was detected. This allows the appropriate notification destination to be selected from among the worker in charge of the work, the instructor, the manager, etc., depending on the worker's status.

[0083] (Embodiment 3) Next, embodiment 3 will be described. In this embodiment, an example is given in which erroneous detection of an abnormality is determined based on the action of the worker. Note that this embodiment can be implemented in combination with embodiment 1 or 2, and each configuration shown in embodiment 1 or 2 may be used as appropriate.

[0084] Fig. 17 shows an example configuration of an anomaly detection device 100 according to some embodiments. In the example of Fig. 17, the anomaly detection device 100 includes a gaze detection unit 180 and an erroneous detection determination unit 190 in addition to the configuration of the example of Fig. 5. Note that the gaze detection unit 180 and the erroneous detection determination unit 190 may be added to the configuration of the example of Fig. 14.

[0085] The gaze detection unit 180 detects the gaze of the worker performing the task in which an abnormality has been detected. For example, the gaze detection unit 180 may estimate the gaze of the worker from the task video using a gaze estimation engine that uses machine learning such as deep learning. The gaze of the worker may be detected not only by the gaze estimation engine but also by a gaze sensor or the like. The gaze detection unit 180 may analyze the worker's movements from the task video, not limited to the gaze of the worker.

[0086] The erroneous detection determination unit 190 determines whether the abnormality detected by the anomaly detection unit 130 is an erroneous detection. The erroneous detection determination unit 190 determines whether the abnormality is an erroneous detection based on the detected line of sight of the worker. Specifically, the erroneous detection determination unit 190 determines whether the worker has viewed the notified abnormality information based on the worker's line of sight. That is, the erroneous detection determination unit 190 determines whether the worker's line of sight is directed toward the display screen of the work display 210 based on the detected line of sight of the worker. For example, if the worker views the work display 210 displaying the abnormality information, or if the worker views the work display 210 for longer than a predetermined time, the erroneous detection determination unit 190 determines that the detected abnormality is not an erroneous detection. Furthermore, if the worker does not view the work display 210 displaying the abnormality information, or if the worker views the work display 210 for a shorter time than the predetermined time, the erroneous detection determination unit 190 determines that the detected abnormality is an erroneous detection. If the worker does not view the abnormality information, the erroneous detection determination unit 190 determines that the abnormality is an erroneous detection because there may be frequent abnormality notifications and the worker may be ignoring the abnormality notifications. The erroneous detection determination unit 190 may determine whether an abnormality has been erroneously detected based on the analysis results of the worker's movements, not just the worker's line of sight.

[0087] If the false detection determination unit 190 determines that the detection is false, the notification unit 160 does not need to notify the notification destination of the detected abnormality. Furthermore, if the false detection determination unit 190 determines that the detection is false, the analysis engine used by the anomaly detection unit 130 to detect the abnormality may be made to label the video of the learning data as false detection and learn the video.

[0088] As described above, a detected abnormality may be determined to be a false positive based on the operator's movements such as line of sight, etc. For example, when the number of learning data is small, the accuracy of the analysis engine decreases, but by determining false positives as in this embodiment, false positives by the analysis engine can be suppressed.

[0089] The present disclosure is not limited to the above-described embodiments, and modifications can be made as appropriate without departing from the spirit of the present disclosure. For example, in the above-described embodiments, abnormalities are detected and notified in real time from images captured by a camera, but abnormalities may be detected and notified at other times, not limited to real time. For example, camera images may be stored, and the images may be analyzed after work, during breaks, during worker changeovers, etc., to detect and notify abnormalities. Detecting and notifying abnormalities as a review of work can be used to evaluate individual work and product quality.

[0090] Each component in the above-described embodiments may be configured with hardware or software, or both, and may be configured with a single piece of hardware or software, or may be configured with multiple pieces of hardware or software. Devices such as an anomaly detection device and each function (process) may be implemented by a computer 30 having a processor 31 such as a CPU (Central Processing Unit) and a memory 32 serving as a storage device, as shown in FIG. 18 . For example, a program for performing the method (selection method) in the embodiment may be stored in the memory 32, and each function may be implemented by the processor 31 executing the program stored in the memory 32.

[0091] These programs include instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0092] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0093] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0094] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A selection system comprising: a detection means for detecting an abnormality in an operation performed by a worker; an acquisition means for acquiring an operation status of the operation; and a selection means for selecting a notification destination to notify of the abnormality detection result based on the acquired operation status. (Supplementary Note 2) The selection system according to Supplementary Note 1, wherein the operation status includes a progress state of the operation. (Supplementary Note 3) The selection system according to Supplementary Note 2, wherein the selection means selects, as a notification destination, a worker who is performing work on a work object for the operation for which the abnormality has been detected. (Supplementary Note 4) The selection system according to any one of Supplements 1 to 3, wherein the operation status includes the status of the worker. (Supplementary Note 5) The selection system according to Supplementary Note 4, wherein the selection means selects, as a notification destination, from among the worker performing the operation for which the abnormality has been detected, an instructor who is instructing the worker, and a manager who manages the operation, according to the status of the worker. (Supplementary Note 6) The selection system according to any one of Supplements 1 to 5, comprising: an analysis means for analyzing the behavior of the worker of the notification destination in response to the abnormality detection result output from the output means of the notification destination; and a determination means for determining whether the abnormality is a false detection in accordance with the analysis result of the worker's behavior. (Supplementary Note 7) A selection device comprising: a detection means for detecting an abnormality in an operation performed by a worker; an acquisition means for acquiring a work status of the operation; and a selection means for selecting a notification destination to notify of the abnormality detection result based on the acquired work status. (Supplementary Note 8) The selection device according to Supplementary Note 7, wherein the work status includes a progress state of the operation. (Supplementary Note 9) The selection device according to Supplementary Note 8, wherein the selection means selects, as a notification destination, a worker who is performing work on a work target object of the operation in which the abnormality was detected. (Supplementary Note 10) The selection device according to any one of Supplements 7 to 9, wherein the work status includes the status of the worker. (Supplementary Note 11) The selection device according to Supplementary Note 10, wherein the selection means selects a notification recipient from among the worker who performed the work in which the abnormality was detected, a trainer who trains the worker, and a manager who manages the work, depending on the status of the worker.(Supplementary Note 12) The selection device according to any one of Supplements 7 to 11, comprising: analysis means for analyzing the behavior of the worker of the notification destination in response to the abnormality detection result output from the output means of the notification destination; and determination means for determining whether the abnormality is erroneously detected based on the analysis result of the worker's behavior. (Supplementary Note 13) A selection method comprising: detecting an abnormality in an operation performed by a worker; acquiring a work status of the operation; and selecting a notification destination to notify of the abnormality detection result based on the acquired work status. (Supplementary Note 14) The selection method according to Supplementary Note 13, wherein the work status includes a progress state of the operation. (Supplementary Note 15) The selection method according to Supplementary Note 14, wherein a worker performing work on a work target of the operation in which the abnormality was detected is selected as the notification destination. (Supplementary Note 16) The selection method according to any one of Supplements 13 to 15, wherein the work status includes the status of the worker. (Supplementary Note 17) The selection method according to Supplementary Note 16, wherein a notification destination is selected from among the worker in charge of the work in which the abnormality was detected, an instructor who is instructing the worker, and a manager who is managing the work, depending on the status of the worker. (Supplementary Note 18) The selection method according to any one of Supplementary Notes 13 to 17, wherein an action of the worker of the notification destination in response to the abnormality detection result output from an output means of the notification destination is analyzed, and a false detection of the abnormality is determined depending on the analysis result of the action of the worker.

[0095] REFERENCE SIGNS LIST 1 Monitoring system 10 Selection system 11 Detection unit 12 Acquisition unit 13 Selection unit 20 Selection device 30 Computer 31 Processor 32 Memory 100 Anomaly detection device 110 Video acquisition unit 120 Sensor data acquisition unit 130 Anomaly detection unit 131 Worker determination unit 132 Work object determination unit 133 Work environment determination unit 140 Progress status determination unit 150 Notification destination selection unit 160 Notification unit 170 Worker information acquisition unit 180 Gaze detection unit 190 False detection determination unit 210 Work display 220 Camera 230 Sensor 310 Management display

Claims

1. A selection system comprising: a detection means for detecting abnormalities in work performed by a worker; an acquisition means for acquiring the work status of the work; and a selection means for selecting a notification destination to notify of the detection result of the abnormality based on the acquired work status.

2. The selection system according to claim 1, wherein the work status includes a progress state of the work.

3. The selection system according to claim 2, wherein the selection means selects, as a notification recipient, a worker who is performing a task on the work object of the task in which the abnormality has been detected.

4. The selection system according to claim 1, wherein the work status includes a status of the worker.

5. The selection system according to claim 4, wherein the selection means selects the notification recipient from among the worker who performed the work that detected the abnormality, the instructor who is instructing the worker, and the administrator who manages the work, depending on the status of the worker.

6. A selection system as claimed in any one of claims 1 to 5, comprising: an analysis means for analyzing the actions of the worker of the notification destination in response to the abnormality detection result output from the output means of the notification destination; and a judgment means for judging whether the abnormality is a false positive depending on the analysis result of the actions of the worker.

7. A selection device comprising: a detection means for detecting an abnormality in work performed by a worker; an acquisition means for acquiring a work status of the work; and a selection means for selecting a notification destination to notify of the detection result of the abnormality based on the acquired work status.

8. The selection device according to claim 7, wherein the work status includes a progress state of the work.

9. The selection device according to claim 8, wherein the selection means selects, as a notification recipient, a worker who is performing a task on a work object of the task in which the abnormality has been detected.

10. The selection device according to any one of claims 7 to 9, wherein the work status includes the status of the worker.

11. The selection device according to claim 10, wherein the selection means selects the notification recipient from among the worker who performed the work in which the abnormality was detected, an instructor who is instructing the worker, and an administrator who manages the work, depending on the status of the worker.

12. A selection device as described in any one of claims 7 to 11, comprising: an analysis means for analyzing the actions of the worker of the notification destination in response to the abnormality detection result output from the output means of the notification destination; and a judgment means for judging whether the abnormality is a false positive depending on the analysis result of the worker's actions.

13. A selection method comprising: detecting an abnormality in work performed by a worker; acquiring a work status of the work; and selecting a notification destination to notify of the detection result of the abnormality based on the acquired work status.

14. The selection method according to claim 13, wherein the work status includes a progress state of the work.

15. The selection method according to claim 14, further comprising selecting, as a notification recipient, a worker who is performing a task on the work object of the task in which the abnormality has been detected.

16. The selection method according to any one of claims 13 to 15, wherein the work status includes the status of the worker.

17. The selection method according to claim 16, further comprising the step of selecting a notification recipient from among the worker who performed the work in which the abnormality was detected, a trainer who trains the worker, and a manager who manages the work, depending on the status of the worker.

18. A selection method according to any one of claims 13 to 17, further comprising: analyzing the actions of the worker of the notification destination in response to the abnormality detection result output from the output means of the notification destination; and determining whether the abnormality is a false positive depending on the analysis result of the worker's actions.

Images