Abnormality cause estimation device
The abnormality cause estimation device uses machine-learned models to analyze skeletal and line-of-sight coordinates for precise work abnormality detection, facilitating swift corrective measures.
Patent Information
- Application Number
- JP2023083866
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-12-04
- Estimated Expiration
- Not applicable · inactive patent
Smart Images

Figure 2024167647000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an abnormality cause estimating device. [Background technology]
[0002] There is a technique for extracting skeletal information of a worker from a video of the worker and generating a video for analyzing the work (for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2021-163293 A Summary of the Invention [Problem to be solved by the invention]
[0004] For example, if an abnormality occurs during work, a supervisor may be able to identify the cause of the abnormality by analyzing the analytical video as described above, but it is difficult to immediately and accurately identify the cause of the abnormality.
[0005] Therefore, the present invention has been made in consideration of the above problems, and has an object to provide an anomaly cause estimating device that can accurately estimate the cause of an abnormality in an operation from video footage of a worker. [Means for solving the problem]
[0006] The above object can be achieved by an anomaly cause estimation device comprising: an extraction unit that extracts skeletal coordinates and gaze coordinates of a worker from a video of the worker; a determination unit that determines whether or not an operation performed by the worker is abnormal based on a time change in the skeletal coordinates and a time change in the gaze coordinates; and an estimation unit that estimates a cause of the abnormality of the operation when the operation is determined to be abnormal, wherein the estimation unit estimates the cause of the abnormality using a machine-learned estimation model through supervised learning that receives the skeletal coordinates and the gaze coordinates as input and outputs the cause of the abnormality. Effect of the Invention
[0007] According to the present invention, it is possible to provide an abnormality cause estimating device that can accurately estimate the cause of an abnormality in an operation from a video of a worker. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a configuration diagram illustrating an example of an abnormality cause estimation system. [Diagram 2] 11 is a flowchart illustrating an example of an abnormality cause estimation control executed by an abnormality cause estimating server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] (Example of schematic configuration of an abnormality cause estimation system) FIG. 1 is a schematic diagram of an anomaly cause estimation system 9. As an example, the anomaly cause estimation system 9 is installed in an assembly plant for automobiles or the like. The anomaly cause estimation system 9 includes an anomaly cause estimation server 1, a camera device 2, and a terminal 4 that are capable of communicating with each other via a LAN (Local Area Network) 90. The anomaly cause estimation server 1 is an example of an anomaly cause estimation device. When an anomaly occurs in the work of a worker, the anomaly cause estimation server 1 estimates the cause of the anomaly. The camera device 2 captures an image of the worker while he or she is working. The terminal 4 is carried by the worker performing the work.
[0010] The anomaly cause estimating server 1 is an example of a computer, and includes a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, a HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, and the communication port 14 via a bus 19 so as to be able to input and output signals to and from each other. The ROM 11 stores a program that drives the CPU 10. The RAM 12 functions as a working memory for the CPU 10. The communication port 14 is, for example, a wireless LAN (Local Area Network) card, and processes communication of the CPU 10 via a LAN 90.
[0011] When the CPU 10 reads the program from the ROM 11, it forms a device control unit 100, an estimation unit 101, a determination unit 102, and an extraction unit 103 as software functions. In addition, the HDD 13 stores worker data 130. The device control unit 100 instructs the estimation unit 101, the determination unit 102, and the extraction unit 103 to operate in accordance with a sequence defined in the program. Note that the device control unit 100, the estimation unit 101, the determination unit 102, and the extraction unit 103 are not limited to software, and may be realized by hardware such as an ASIC (Application Specified Integrated Circuit).
[0012] The extraction unit 103 extracts the skeletal coordinates and gaze coordinates of the worker by image analysis from the video captured by the camera device 2. The extraction unit 103 receives the video from the camera device 2 via the LAN 90. The extraction unit 103 may extract the skeletal coordinates and gaze coordinates of the worker by image analysis using, for example, a general-purpose AI (Artificial Intelligence). The determination unit 102 determines whether the work of the worker is abnormal or not based on the time change of the skeletal coordinates and the time change of the gaze coordinates extracted by the extraction unit 103. The estimation unit 101 estimates the cause of the abnormality of the work, which will be described in detail later. The work is, for example, part removal work, part assembly work, part removal work, moving work, welding work, painting work, inspection work, product transportation work, etc.
[0013] The extraction unit 103 stores a time series of coordinates of multiple points of the worker's skeleton and gaze coordinates in the HDD 13 as worker data 130. The worker data 130 indicates, for example, the coordinates of the worker's nose, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip, as well as the worker's gaze coordinates, for each frame image number. The coordinates are indicated by X and Y coordinates (x, y), for example, when the horizontal and vertical directions of a rectangular frame of the image are defined as the X and Y axes, respectively. There is no limitation on the position of the origin.
[0014] (Anomaly cause estimation control) 2 is a flowchart illustrating an example of anomaly cause estimation control executed by the anomaly cause estimating server 1. First, the extracting unit 103 receives video data from the camera device 2 via the communication port 14 (step S1). The extracting unit 103 receives the video data as a plurality of frame images. Next, the extracting unit 103 extracts the skeleton coordinates and gaze coordinates of the worker from each frame image (step S2). Next, the extracting unit 103 stores the extraction results as worker data 130 in the HDD 13 (step S3).
[0015] The judgment unit 102 judges whether an abnormality has occurred in the work of the worker based on the worker data 130 (step S4). This judgment is performed, for example, as follows. The judgment unit 102 calculates the amount of displacement of the skeleton coordinates and the amount of displacement of the line-of-sight coordinates from a predetermined number of consecutive frame images. The judgment unit 102 judges whether an abnormality has occurred in the work based on the comparison result of comparing the amount of displacement of the skeleton coordinates with a predetermined threshold and the comparison result of comparing the amount of displacement of the line-of-sight coordinates with a predetermined threshold. For example, when at least one of the amount of displacement of the skeleton coordinates and the amount of displacement of the line-of-sight coordinates is equal to or greater than the corresponding threshold, the judgment unit 102 judges that an abnormality has occurred in the work. Examples of actions that are judged to have caused an abnormality in the work include an action of the worker dropping a part and taking a new part out of the parts box, an action of picking up the dropped part, an action of throwing a part into a defective product box, and an action of making a mistake in the process order. If the answer is No in step S4, the control ends.
[0016] If the answer is Yes in step S4, the estimation unit 101 estimates the cause of the work abnormality using an estimation model based on the skeleton coordinates and gaze coordinates in the frame image where the abnormality occurred and the skeleton coordinates and gaze coordinates in the frame image immediately before the abnormality occurred (step S5). The estimation model is a model that has been machine-learned by supervised learning. By inputting the above skeleton coordinates and gaze coordinates to the estimation model, the estimation model outputs the cause of the abnormality. Examples of the cause of the abnormality include the worker's deviation from the line of sight, poor posture of the worker, poor position of the parts box, poor arrangement order of the parts box, abnormal work speed, and excessive work load. A neural network is used for supervised learning. In this way, the estimation model that has been machine-learned by supervised learning is used to accurately estimate the cause of the work abnormality. In addition, the cause of the work abnormality is estimated based on not only the skeleton coordinates and gaze coordinates at the time of the abnormality occurrence but also the skeleton coordinates and gaze coordinates in the normal state immediately before the occurrence. As a result, for example, an error in the work order or the worker's movement can be accurately identified, and the cause of the abnormality can be accurately estimated. It should be noted that the number of frame images in which an abnormality has occurred and the number of frame images immediately before the occurrence of an abnormality are not limited to one, and each may be a predetermined number.
[0017] The estimation unit 101 judges whether the cause of the abnormality in the work has been identified (step S6). For example, if there is one output value that is equal to or greater than a predetermined value among the output values corresponding to the multiple abnormality causes of the estimation model, it is judged that the cause of the abnormality has been identified. If there is no output value that is equal to or greater than a predetermined value among the output values corresponding to the multiple abnormality causes of the estimation model, it is judged that the cause of the abnormality has not been identified.
[0018] If the answer is Yes in step S6, the device control unit 100 causes the worker's terminal 4 to display the details of the abnormality in the work and the cause of the abnormality via the LAN 90 (step S7). This allows the worker or supervisor to easily understand the work's improvements and problems after an abnormality occurs, and to take immediate measures. If the answer is No in step S6, the device control unit 100 causes the worker's terminal 4 to display the details of the abnormality in the work and the fact that the cause of the abnormality is unknown via the LAN 90 (step S8). The device control unit 100 may store the work details and the cause of the abnormality in the HDD 13 in association with each other.
[0019] Step S5 may be executed when the number of times it is determined in step S4 that an abnormality has occurred in the work exceeds a threshold value. The worker's posture and height may be added to the worker data in addition to the worker's skeleton coordinates and line of sight coordinates. The worker's posture and height may be extracted from a video of the worker. The worker's height may be input to the anomaly cause estimating server 1 from the terminal 4 via the LAN 90, for example.
[0020] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this embodiment and can be modified in various ways without departing from the scope of the present invention. [Explanation of symbols]
[0021] 1. Anomaly cause estimation server (anomaly cause estimation device) 2 Camera equipment 10 CPU 101 Estimation part 102 Judgment section 103 Extraction part
Claims
[Claim 1] an extraction unit that extracts skeletal coordinates and gaze coordinates of a worker from a video of the worker; a determination unit that determines whether or not the work performed by the worker is abnormal based on the time change of the skeleton coordinates and the time change of the gaze coordinate; and an estimation unit that estimates a cause of the abnormality of the work when the work is determined to be abnormal, The estimation unit estimates the cause of the abnormality using a machine-learned estimation model through supervised learning that receives the skeleton coordinates and gaze coordinates as input and outputs the cause of the abnormality.
Citation Information
Patent Citations
Work analyzer and work analysis program
JP2021163293A
