Work evaluation device

Abstract

To provide a work evaluation device capable of properly evaluating the work of an operator.SOLUTION: A work evaluation device includes: a detection unit that detects coordinates of bones of an operator from an image obtained by imaging the work of the operator; a determination that determines an abnormality in the work on the basis of the coordinates of the bones of the operator when the work is normal, from the coordinates of the bones of the operator detected by the detection unit; a calculation unit that calculates a frequency at which the abnormality is detected; and an evaluation unit that evaluates the work from the frequency and the degree of the abnormality based on equipment or a component related to the abnormality.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a work evaluation device.

Background Art

[0002] For example, Patent Document 1 describes determining an unsafe action by discriminating the actions of an operator and comparing them with the basic action pattern.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since the above method simply determines whether the actions of the operator are safe, it cannot appropriately evaluate the work of the operator and is difficult to utilize for improving the work ability of the operator.

[0005] Therefore, the present invention has been made in view of the above problems, and an object thereof is to provide a work evaluation device that can appropriately evaluate the work of an operator.

Means for Solving the Problems

[0006] The work evaluation device of the present invention includes a detection unit that detects the coordinates of the skeleton of the operator from an image obtained by imaging the work of the operator, a determination unit that determines an abnormality of the work based on the coordinates of the skeleton of the operator detected by the detection unit and based on the coordinates of the skeleton of the operator when the work is normal, a calculation unit that calculates the frequency with which the abnormality is determined, and an evaluation unit that evaluates the work from the frequency and the degree of the abnormality based on the equipment or members related to the abnormality.

[0007] In the above-described work evaluation device, the discrimination unit may determine that the abnormality is an action in which the worker drops the equipment or the component, and the evaluation unit may evaluate the work based on the degree of the abnormality, which is determined by the mass of the equipment or component that fell in the action.

[0008] In the above-described work evaluation device, the discrimination unit may determine that the abnormality is an action in which the worker drops the equipment or component, and the evaluation unit may evaluate the work based on the degree of the abnormality, which is determined by the distance between the equipment or component that fell in the action and the coordinates of the worker's skeleton.

[0009] In the above-described work evaluation device, the discrimination unit may determine that the worker approaches the equipment as an abnormality, and the evaluation unit may evaluate the work using the distance between the equipment and the coordinates of the worker's skeleton as the degree of the abnormality.

[0010] In the above-described work evaluation device, the evaluation unit may calculate the evaluation value of the work using a supervised learning-based computational model that takes the coordinates of the worker's skeleton in the image, the coordinates of the equipment or component in the image, and the frequency as inputs and outputs the evaluation value of the work. [Effects of the Invention]

[0011] According to the present invention, the work of an employee can be appropriately evaluated. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a diagram illustrating an example of a work evaluation system. [Figure 2] Figure 2 shows an example of an image from a camera device that captures a worker performing normal tasks. [Figure 3] Figure 3 shows an example of an image from a camera device that captured the action of a worker dropping a part. [Figure 4]Figure 4 shows an example of an image from a camera device that captured the movement of worker H approaching the press machine. [Figure 5] Figure 5(a) shows an example of evaluation values ​​when using an abnormality score based on the mass of the parts and tools. Figure 5(b) shows an example of evaluation values ​​when using an abnormality score based on the distance between the worker and the parts and tools. Figure 5(c) shows an example of evaluation values ​​when using an abnormality score based on the distance between the worker and the machine. [Figure 6] Figure 6 is a flowchart showing an example of the evaluation process of the work evaluation server. [Modes for carrying out the invention]

[0013] (Example of a work evaluation system configuration) Figure 1 is a configuration diagram showing an example of the work evaluation system 9. The work evaluation system 9 is installed, for example, in an assembly plant for automobiles or other products.

[0014] The work evaluation system 9 includes a work evaluation server 1, a camera device 2, and a terminal 3 that can communicate with each other via a LAN (Local Area Network) 90. The work evaluation server 1 is an example of a work evaluation device. The work evaluation server 1 evaluates the worker's work. The camera device 2 captures images of the worker. The terminal 3 is held by, for example, the worker or their associates, and receives and notifies them of the work evaluation results from the work evaluation server 1.

[0015] The work evaluation server 1 has a CPU (Central Processing Unit) 10, ROM (Read Only Memory) 11, RAM (Random Access Memory) 12, HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, RAM 12, HDD 13, and communication port 14 via a bus 19 so that they can input and output signals to and from each other.

[0016] The ROM 11 stores a program for driving the CPU 10. The RAM 12 functions as a working memory of the CPU 10. The communication port 14 is, for example, a wireless LAN (Local Area Network) card and processes the communication of the CPU 10 via the LAN 90.

[0017] When the CPU 10 reads a program from the ROM 11, it forms, as software functions, a device control unit 100, a skeleton detection unit 101, an operation determination unit 102, an abnormal frequency calculation unit 103, and a work evaluation unit 104. Further, the HDD 13 stores detected skeleton data 130, reference skeleton data 131, equipment data 132, and evaluation data 133. The device control unit 100 instructs operations to the skeleton detection unit 101, the operation determination unit 102, the abnormal frequency calculation unit 103, and the work evaluation unit 104 according to the sequence defined in the program. Note that the device control unit 100, the skeleton detection unit 101, the operation determination unit 102, the abnormal frequency calculation unit 103, and the work evaluation unit 104 are not limited to software and may be realized by hardware such as an ASIC (Application Specified Integrated Circuit).

[0018] The skeleton detection unit 101 is an example of a detection unit. The skeleton detection unit 101 detects the coordinates of the operator's skeleton from an image obtained by imaging the operator's work with the camera device 2. The skeleton detection unit 101 receives image data from the camera device 2 via the LAN 90. The skeleton detection unit 101 detects the coordinates of the operator's skeleton, for example, by image analysis using general-purpose AI (Artificial Intelligence).

[0019] For example, the skeleton detection unit 101 stores the coordinates of multiple locations of the operator's skeleton as time-series detected skeleton data 130 in the HDD 13. The detected skeleton data 130 indicates the coordinates of the nose, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left waist, and right waist of each person #1 to #N (N: positive integer) in the image for each frame number of the image, for example. The coordinates are represented by X coordinates and Y coordinates (x, y) when the horizontal and vertical directions of the rectangular frame of the image are defined as the X axis and the Y axis, respectively. Note that the position of the origin is not limited.

[0020] The operation determination unit 102 is an example of a determination unit. The operation determination unit 102 determines the abnormality of the operation based on the coordinates of the operator's skeleton detected by the skeleton detection unit 101 and based on the coordinates of the operator's skeleton when the operation is normal. The operation determination unit 102 compares the detected skeleton data 130 with the reference skeleton data 131. The reference skeleton data 131 shows the coordinates of each part of the operator's skeleton performing a formal work operation in time series in the same manner as the detected skeleton data 130. For example, when the operation determination unit 102 compares the detected skeleton data 130 and the reference skeleton data 131, the operation determination unit 102 determines the abnormality of the operation from the number of consecutive frames, that is, the duration, in which the average value of the differences in the coordinates of the corresponding skeletons is greater than or equal to a predetermined threshold.

[0021] The abnormality of the operation is an operation different from the formal work operation shown in the reference skeleton data 131. In this example, as the abnormality of the operation, an operation in which the operator drops a tool or part used in the operation and an operation in which the operator approaches the machine are cited, but it is not limited thereto. Note that the tool and the machine are examples of equipment, and the part is an example of a member.

[0022] The abnormality frequency calculation unit 103 is an example of a calculation unit. The abnormality frequency calculation unit 103 calculates the frequency (hereinafter referred to as the abnormality frequency) with which the operation determination unit 102 determines the abnormality of the operation. The abnormality frequency is, for example, the ratio of the number of unit operations determined to be abnormal to the total number of unit operations performed by the operator, but is not limited thereto, and may be, for example, the ratio of the unit operations determined to be abnormal to the total work time.

[0023] The work evaluation unit 104 is an example of an evaluation unit. The work evaluation unit 104 evaluates the work based on the abnormality frequency calculated by the abnormality frequency calculation unit 103 and the degree of abnormality (hereinafter referred to as the degree of abnormality) related to the tools, parts, or machinery involved in the work abnormality, as determined by the operation determination unit 102. Therefore, the work evaluation unit 104 can appropriately evaluate the accuracy of the worker's handling of tools, parts, or machinery, compared to, for example, simply evaluating whether the work is safe or not.

[0024] The degree of abnormality is determined by the magnitude of the impact that a tool, part, or machine has on the work. For example, if the motion determination unit 102 detects that a worker has dropped a tool or part, the work evaluation unit 104 calculates the degree of abnormality based on the weight of the dropped tool or part, or the distance between the tool or part and the coordinates of the worker's skeleton. Also, if the motion determination unit 102 detects that a worker has approached a machine, the work evaluation unit 104 calculates the degree of abnormality based on the distance between the machine and the coordinates of the worker's skeleton.

[0025] The work evaluation unit 104 calculates an evaluation value for the work based on the frequency and severity of abnormalities and records it as evaluation data 133. For example, when all work is completed, the work evaluation unit 104 transmits the evaluation data 133 to terminal 3 via LAN 90. This allows the evaluation data 133 to be used to improve the work performance of the workers.

[0026] (Example of normal operation) Figure 2 shows examples of images G1a and G2a from camera device 2, which captured images of worker H performing normal work. The X and Y axes in images G1a and G2a correspond to the horizontal and vertical directions of the paper, respectively. This is also the case in subsequent drawings.

[0027] Images G1a and G2a from camera device 2 capture worker H in the workshop. The workshop includes, for example, a workbench 80 where work is performed, a parts rack 81 for storing parts C, a conveyor belt 83, and a press machine 82. Here, the press machine 82 is just one example of the above-mentioned machinery. Camera device 2 is positioned directly in front of worker H as he faces the workbench 80.

[0028] In this example, the task is described as the worker assembling part C, which has been taken from the parts shelf 81, onto the workpiece W, but it is not limited to this. Other tasks may include painting part C, welding part C to the workpiece W, and inspecting part C.

[0029] The motion determination unit 102 detects the coordinates of the worker H's skeleton from images G1a and G2a. Examples of the skeletal coordinates include the position of worker H's nose P0, left shoulder P23, right shoulder P13, left elbow P22, right elbow P12, left hand P21, right hand P11, left hip P32, and right hip P31. Note that positions P0, P11-P13, P21-P23, P31, and P32 are shown only in image G1a and are omitted from images G2a and later.

[0030] The motion determination unit 102 identifies the worker H's work movements based on the time changes in the coordinates of each part of the skeleton. Image G1a shows worker H holding the workpiece W with his left hand and aligning the assembly position of part C with his right hand, and image G2a shows worker H assembling part C to the workpiece W.

[0031] The motion determination unit 102 determines normal work operation by comparing the time-varying patterns of the coordinates of skeletons in images G1a, G2a, and other related images with the time-varying patterns of the coordinates of skeletons in the reference skeleton data 131. For example, the motion determination unit 102 determines normal work operation if, for example, the average value of the differences in the coordinates of each corresponding skeleton between the detected skeleton data 130 and the reference skeleton data 131 is greater than or equal to a threshold for a predetermined number of consecutive frames. Here, the difference in coordinates is the distance between coordinates in the XY plane.

[0032] (Example of an action that drops a part) Figure 3 shows an example of an image from camera device 2 capturing the action of worker H dropping part C. In Figure 3, components common to Figure 2 are denoted by the same reference numerals, and their explanations are omitted. The image shows worker H accidentally dropping part C onto the conveyor belt 83 when attempting to assemble part C onto workpiece W, as shown in image G1a.

[0033] The motion determination unit 102 determines abnormalities in work by comparing the time-dependent changes in the coordinates of skeletons in the above image and other related images with the time-dependent changes in the coordinates of skeletons in the reference skeleton data 131. For example, the motion determination unit 102 determines an abnormal work operation if the number of consecutive frames in which the average value of the differences in the coordinates of each corresponding skeleton in the detected skeleton data 130 and the reference skeleton data 131 is greater than or equal to a threshold is greater than or equal to a predetermined number.

[0034] In this case, for example, the motion determination unit 102 determines, based on the time change of the skeleton coordinates in the detected skeleton data 130, that the action of the worker reaching out with their right hand to pick up the dropped part C is an abnormal operation. The abnormality frequency calculation unit 103 then adds one to the number of abnormal operations and calculates the number of abnormal operations relative to the total number of operations as the abnormality frequency.

[0035] The work evaluation unit 104 calculates the degree of abnormality based on part C from the image. The degree of abnormality is based on the weight of part C or the distance La between part C and worker H. The equipment data 132 contains the names of equipment such as part C and tools (bolt, screwdriver, pliers, etc.), their image data, and weight.

[0036] The work evaluation unit 104 extracts an image of part C from the image data of the equipment data 132, for example, identifies the name of part C, and calculates its weight as the degree of abnormality. In this case, the work evaluation unit 104 evaluates the work based on the degree of abnormality derived from the weight of part C. Therefore, it becomes possible to evaluate the work while considering, for example, the possibility of work interruption due to part C falling and being damaged, or the possibility of work interruption due to part C colliding with and damaging another object when it falls.

[0037] Furthermore, the work evaluation unit 104 can, for example, extract an image of part C from the image data of the equipment data 132 and calculate the degree of abnormality from the distance La between the coordinates of part C in the image and the skeleton of worker H. In this case, the distance La is, for example, the difference between the X coordinate of the end of part C on the worker H side and the X coordinate of worker H's nose, but is not limited to this. The work evaluation unit 104 evaluates the work based on the degree of abnormality derived from the distance La. Therefore, for example, it becomes possible to evaluate the possibility of work interruption due to part C coming into contact with worker H when it falls. Thus, in this example, the impact on the work due to part C falling is calculated as the degree of abnormality.

[0038] In this example, part C was given as an example of a fallen component, but the example is not limited to this. The degree of abnormality can be calculated in the same way for an action in which a tool such as a screwdriver falls instead of part C. Furthermore, in calculating the distance La, any coordinate of the worker H's skeleton may be used, and the degree of abnormality may be varied depending on the type of skeleton used in the calculation (nose, right hand, right shoulder, etc.). In addition, the work evaluation unit 104 may evaluate the work using the weight of part C or the tool and the distance La.

[0039] (Example of an action that approaches a machine) Figure 4 shows an example of an image from the camera device 2 capturing the action of worker H approaching the press machine 82. In Figure 4, components common to both Figure 2 and Figure 2 are denoted by the same reference numerals, and their explanations are omitted.

[0040] The image shows worker H attempting to assemble part C to workpiece W as shown in image G1a, but instead of adhering to the normal assembly posture as shown in image G2a above, approaching the press machine 82 and attempting to assemble part C to workpiece W in an inappropriate posture. The press machine 82 is an example of equipment that worker H's approach to is restricted, but the equipment is not limited to this, and could also include precision equipment that could be damaged by careless contact with worker H.

[0041] As described above, the motion determination unit 102 determines work abnormalities from the difference in the coordinates of each skeleton between the detected skeleton data 130 and the reference skeleton data 131. For example, the motion determination unit 102 determines that the movement of worker H approaching the press machine 82 is a work abnormality based on the time change of the skeleton coordinates in the detected skeleton data 130. The abnormality frequency calculation unit 103 adds one to the number of work abnormalities and calculates the number of work abnormalities relative to the total number of operations as the abnormality frequency.

[0042] The work evaluation unit 104 calculates an abnormality score based on the press machine 82 from the image. The abnormality score is based, for example, on the distance Lb between the press machine 82 and the worker H. In this case, the distance Lb is, for example, the difference between the X coordinate of the end of the press machine 82 on the worker H side and the X coordinate of the worker H's nose, but is not limited to this. The work evaluation unit 104 evaluates the work based on the abnormality score based on the distance Lb. This makes it possible to evaluate the work while considering the possibility of work interruption due to the worker H coming into contact with the press machine 82. In addition, any coordinate of the worker H's skeleton may be used in calculating the distance Lb, and furthermore, the abnormality score may be different depending on the type of skeleton used in the calculation (nose, right hand, right shoulder, etc.). Also, the work evaluation unit 104 may calculate an abnormality score based on the press machine 82 only when the press machine 82 is in operation.

[0043] (Example of calculation of evaluation value) Figure 5(a) shows an example of an evaluation value when using an abnormality score based on the mass of parts and tools. In this case, the work evaluation unit 104 calculates an evaluation value from "1" to "5" from the mass and the frequency of abnormalities. Here, the evaluation value is calculated each time an abnormality in the work is detected, and the higher the value, the better the overall work is performed. In this example, the higher the frequency of abnormalities and the greater the mass, the lower the evaluation value. Therefore, the more frequently a worker drops heavy parts and tools, the lower the evaluation will be.

[0044] Figure 5(b) shows an example of an evaluation value when using an anomaly score based on the distance between the worker and the parts / tools. In this example, the higher the frequency of anomalies and the shorter the distance, the lower the evaluation value. Therefore, the more frequently a worker drops parts / tools near themselves, the lower their evaluation will be.

[0045] Figure 5(c) shows an example of an evaluation value when using an anomaly score based on the distance between the worker and the machine. In this example, the higher the frequency of anomalies and the shorter the distance, the lower the evaluation value. Therefore, the more frequently the worker approaches the machine, the lower the evaluation will be.

[0046] AI may be used to calculate the evaluation value. In this case, the work evaluation unit 104 takes the coordinates of the worker's skeleton in the image of the camera device 2, the coordinates of the machine, tool, or part in the image, and the frequency of anomalies as input and calculates the work evaluation value using a supervised learning-based calculation model that outputs the work evaluation value.

[0047] Specifically, in the learning process of the computational model, the evaluation value of the work is annotated based on the time change of the coordinates of the skeletal structure of the training data, the coordinates of the machine, tool, or part, and the frequency of anomalies. The computational model is a neural network that mathematically models the function of the human brain, and the weight coefficients of the activation function of the parts corresponding to neurons are determined based on machine learning. In this way, the work evaluation unit 104 can calculate the evaluation value with high accuracy by using a computational model that has been trained through supervised learning.

[0048] (Evaluation process) Figure 6 is a flowchart showing an example of the evaluation process of the work evaluation server 1. This process is executed by the CPU when a worker starts an example task, for example, in response to instructions from terminal 3.

[0049] First, the skeleton detection unit 101 receives image data from the camera device 2 via the communication port 14 (step St1). The skeleton detection unit 101 receives images of each of the above actions on a frame-by-frame basis as image data. Next, the skeleton detection unit 101 detects the coordinates of the worker's skeleton from the images (step St2). Next, the skeleton detection unit 101 saves the detection result for each frame of the image as detected skeleton data 130 in the HDD 13 (step St3).

[0050] Next, the motion determination unit 102 compares the coordinates of each skeleton in the detected skeleton data 130 and the reference skeleton data 131 frame by frame (step St4). Next, the motion determination unit 102 compares the number of consecutive frames in which the average value of the difference between the coordinates of each skeleton in the detected skeleton data 130 and the reference skeleton data 131 is greater than or equal to a predetermined threshold (hereinafter referred to as the number of differing frames) with a predetermined number N (step St5). If the number of differing frames is less than or equal to the predetermined number N (No in step St5), the motion determination unit 102 determines that the operation is normal (step St12).

[0051] Furthermore, if the number of differing frames is greater than a predetermined number N (Yes in step St5), the operation determination unit 102 determines the abnormality of the operation as described above from the coordinates of each skeleton in the detected skeleton data 130 (step St6). Next, the abnormality frequency calculation unit 103 calculates the abnormality frequency (step St7). Next, the work evaluation unit 104 calculates the degree of abnormality using a method corresponding to the abnormal operation, as described above (step St8). Next, the work evaluation unit 104 calculates an evaluation value from the abnormality frequency and degree of abnormality as described above (step St9). The evaluation value is recorded in the HDD 13 as evaluation data 133.

[0052] Next, the device control unit 100 determines whether all work has been completed based on the predetermined work process (step St10). If work remains (No in step St10), each process from step St1 onwards is executed again. If all work has been completed (Yes in step St10), the device control unit 100 sends evaluation data 133 to terminal 3 via LAN 90 (step St11). In this way, the evaluation process of the work evaluation server 1 is executed.

[0053] The embodiments described above are preferred examples of the present invention. However, the invention is not limited thereto, and various modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0054] 1 Work evaluation server (work evaluation device), 10 CPU, 101 Skeleton detection unit (detection unit), 102 Operation discrimination unit (discrimination unit), 103 Anomaly frequency calculation unit (calculation unit), 104 Work evaluation unit (evaluation unit), 130 Detected skeleton data, 131 Reference skeleton data, 133 Evaluation data

Claims

1. A detection unit that detects the coordinates of the worker's skeleton from an image captured of the worker's work, A determination unit that determines whether the work is abnormal based on the coordinates of the worker's skeleton when the work is normal, from the coordinates of the worker's skeleton detected by the detection unit, A calculation unit that calculates the frequency at which the aforementioned abnormality is detected, The system includes an evaluation unit that evaluates the work based on the frequency and the degree of the abnormality based on the equipment or component involved in the abnormality. Work evaluation device.

2. The discrimination unit determines that the abnormality is an action in which the worker dropped the equipment or the component. The evaluation unit evaluates the operation based on the degree of abnormality, which is determined by the mass of the equipment or component that fell during the operation. The work evaluation device according to claim 1.

3. The discrimination unit determines that the abnormality is an action in which the worker dropped the equipment or the component. The evaluation unit evaluates the work based on the degree of abnormality, which is determined by the distance between the equipment or component that fell during the operation and the coordinates of the worker's skeleton. The work evaluation device according to claim 1.

4. The discrimination unit determines that the abnormality is the action of the worker approaching the equipment. The evaluation unit evaluates the work using the distance between the equipment and the coordinates of the worker's skeleton as the degree of the abnormality. The work evaluation device according to claim 1.

5. The evaluation unit takes the coordinates of the worker's skeleton in the image, the coordinates of the equipment or component in the image, and the frequency as inputs, and calculates the evaluation value of the work using a supervised learning-based computational model that outputs the evaluation value of the work. A work evaluation device according to any one of claims 1 to 4.

Images