Work evaluating apparatus
The work evaluation device addresses inaccuracies in worker movement evaluation by aligning skeletal coordinates using a learning model to correct for camera position deviations, ensuring accurate work assessment.
Patent Information
- Application Number
- JP2023100655
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-06-20
- Publication Date
- 2025-01-08
AI Technical Summary
Existing work evaluation systems face inaccuracies due to misalignment between the camera position during image capture and the learning model's baseline, leading to incorrect evaluation of worker movements.
A work evaluation device that detects and corrects skeletal coordinates using a learning model to align body axes, ensuring accurate evaluation by adjusting for deviations caused by camera position changes.
Enables high-accuracy evaluation of worker movements by correcting skeletal deviations, thereby improving the reliability of work assessment.
Smart Images
Figure 2025001195000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an activity evaluation device. [Background technology]
[0002] For example, Patent Document 1 describes a service that recognizes human behavior from camera footage based on skeletal information, using a learning model in which basic movements have already been learned according to machine learning algorithms such as deep learning. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2022-3434 Summary of the Invention [Problem to be solved by the invention]
[0004] However, for example, when evaluating work using a learning model such as the one described above based on the skeleton of a worker in an image, there is a risk that the worker's movements will not be evaluated accurately because the position of the camera device when imaging the worker may deviate from the position when imaging the person performing the basic movements that are the subject of the learning model.
[0005] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide an activity evaluation device that can evaluate activities with high accuracy. [Means for solving the problem]
[0006] The work evaluation device of the present invention includes a detection unit that detects the skeletal coordinates of a first worker in a first image captured by an imaging device and the skeletal coordinates of a second worker in a second image captured by the imaging device; an evaluation unit that uses a learning model that has been machine-learned to learn the skeletal coordinates of the first worker while working in the first image according to a time series, and evaluates the work of the second worker based on changes over time in the skeletal coordinates of the second worker while working in the second image; a calculation unit that calculates a deviation between the body axes of the first worker and the second worker from the skeletal coordinates of the first worker while working in the first image and the skeletal coordinates of the second worker while working in the second image; and a correction unit that corrects the skeletal coordinates of the first worker learned by the learning model based on the deviation between the body axes. Effect of the Invention
[0007] According to the present invention, work can be evaluated with high accuracy. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a configuration diagram showing an example of a task evaluation system. [Diagram 2] FIG. 2 is a diagram illustrating an example of correction of learning skeleton data. [Diagram 3] FIG. 3 is a flowchart showing an example of the operation of the activity evaluation server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] (Example of the configuration of a work evaluation system) 1 is a configuration diagram showing an example of a work evaluation system 9. As an example, the work evaluation system 9 is installed in an assembly plant for automobiles or the like.
[0010] The work evaluation system 9 includes a work evaluation server 1, a camera device 2, and a terminal 3 which are capable of communicating with each other via a LAN (Local Area Network) 90. The work evaluation server 1 is an example of a work evaluation device. The work evaluation server 1 evaluates the work actions of a worker. The camera device 2 is an example of an imaging device, and captures images of people such as workers. The terminal 3 is carried, for example, by the worker or a related person, and receives and notifies the work evaluation results from the work evaluation server 1.
[0011] The work evaluation server 1 has a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, a HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, and the communication port 14 via a bus 19 so as to be able to input and output signals among them.
[0012] The ROM 11 stores a program that drives the CPU 10. The RAM 12 functions as a working memory for the CPU 10. The communication port 14 is, for example, a wireless LAN (Local Area Network) card, and processes communication of the CPU 10 via a LAN 90.
[0013] When the CPU 10 reads a program from the ROM 11, it forms, as software functions, a device control unit 100, a skeleton detection unit 101, an axis deviation calculation unit 102, a skeleton data correction unit 103, and an operation evaluation unit 104. In addition, the HDD 13 stores detected skeleton data 130, learned skeleton data 131, evaluation data 132, and a learning model 133.
[0014] The device control unit 100 instructs the skeleton detection unit 101, axis deviation calculation unit 102, skeleton data correction unit 103, and task evaluation unit 104 to operate in accordance with a sequence defined in a program. Note that the device control unit 100, skeleton detection unit 101, axis deviation calculation unit 102, skeleton data correction unit 103, and task evaluation unit 104 are not limited to being realized by software, and may be realized by hardware such as an ASIC (Application Specified Integrated Circuit).
[0015] The skeleton detection unit 101 is an example of a detection unit. The skeleton detection unit 101 detects the coordinates of the skeleton of a person in an image captured by the camera device 2. The skeleton detection unit 101 receives image data from the camera device 2 via the LAN 90. The skeleton detection unit 101 detects the coordinates of the skeleton of the person by image analysis using, for example, a general-purpose AI (Artificial Intelligence).
[0016] When the task evaluation unit 104 evaluates a worker to be evaluated (hereinafter referred to as the evaluation worker), the camera device 2 captures an image of the evaluation worker. In addition, the camera device 2 captures an image of an exemplary worker (hereinafter referred to as the exemplary worker) in order to allow the learning model 133 used by the task evaluation unit 104 for evaluation to learn the coordinates of the skeleton of the evaluation standard. Note that the exemplary worker is an example of a first worker, and the evaluation worker is an example of a second worker. Also, the image captured of the exemplary worker is an example of the first image, and the image captured of the evaluation worker is an example of the second image.
[0017] The skeleton detection unit 101 detects the coordinates of the evaluation worker's skeleton from the image of the evaluation worker and stores them in the HDD 13 as detected skeleton data 130. The detected skeleton data 130 indicates, for example, the coordinates of the evaluation worker's nose, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip in the image in chronological order for each frame number of the image. The coordinates are indicated by the X coordinate and the Y coordinate (x, y) when the horizontal and vertical directions of the rectangular frame of the image are defined as the X axis and the Y axis, respectively. There is no limitation on the position of the origin.
[0018] The work evaluation unit 104 is an example of an evaluation unit. The work evaluation unit 104 uses a learning model 133 that has been machine-learned according to a time series of the skeletal coordinates of the model worker working in the image, and evaluates the work of the evaluation worker based on the time change in the skeletal coordinates of the evaluation worker working in the image. The work evaluation unit 104 determines whether the time change in the skeletal coordinates of the evaluation worker recorded in the detected skeleton data 130 is similar to the time change in the skeletal coordinates of the evaluation worker recorded in the learned skeleton data 131, thereby determining whether the evaluation worker is working in accordance with a predetermined standard indicated by the model worker.
[0019] The learning model 133 is stored in the HDD 13 as a set of various calculation modules and parameters. In the learning process of the learning model 133, the time change in the coordinates of the skeleton of the model worker during the work is used as teacher data to annotate the evaluation result of the work. The learning model 133 is a neural network that mathematically models the function of the human brain, and determines the weight coefficient of the activation function of the part corresponding to the neuron based on machine learning. In this way, the work evaluation unit 104 can output the evaluation result with high accuracy by using the learning model 133 that has been machine-learned by supervised learning.
[0020] The coordinates of the skeleton of the model worker learned by the learning model 133 are stored in the HDD 13 as learned skeleton data 131. Similar to the detected skeleton data 130, the learned skeleton data 131 indicates the coordinates of the nose, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip of the evaluation worker in the image in chronological order for each frame number of the image. The learning model 133 outputs an evaluation result based on the degree of difference between the learned skeleton data 131 and the detected skeleton data 130.
[0021] In this way, the work evaluation unit 104 determines the evaluation result as "good" if the evaluator's work complies with the predetermined criteria, and determines the evaluation result as "fail" if the evaluator's work does not complies with the predetermined criteria. The work evaluation unit 104 saves the evaluation result as evaluation data 132 in the HDD 13. The work evaluation unit 104 transmits the evaluation data 132 to the terminal 3 via the LAN 90. This notifies the evaluator of the evaluation result of the work, and the evaluator's work is improved.
[0022] The axis deviation calculation unit 102 is an example of a calculation unit. The axis deviation calculation unit 102 calculates the deviation between the body axes of the model worker and the evaluation worker from the skeletal coordinates of the evaluation worker working in the image of the camera device 2 and the skeletal coordinates of the model worker working in the image of the camera device 2. Here, the body axis is the main axis (central axis in the longitudinal direction) of the torso of each of the model worker and the evaluation worker, and is defined as a straight line passing through, for example, the center point of the coordinates of the skeletons of the right shoulder and the left shoulder and the center point of the coordinates of the right hip and the left hip.
[0023] The skeletal data correction unit 103 is an example of a correction unit. The skeletal data correction unit 103 corrects the coordinates of the skeleton of the exemplary worker learned by the learning model 133 based on the deviation of the body axes of the exemplary worker and the evaluation worker. For example, the skeletal data correction unit 103 calculates the difference in the X coordinates of the body axes as the deviation of the body axes. The skeletal data correction unit 103 corrects the coordinates of the skeleton of the learning skeletal data 131 based on the deviation of the body axes so that the X coordinates of the body axes substantially match.
[0024] Therefore, even if the position of the camera device 2 when the evaluation worker is imaged shifts from the position when the model worker is imaged for learning the learning model 133, the coordinates of the skeleton in the learning skeleton data 131 can be aligned with the position of the evaluation worker in the image of the shifted camera device 2. This allows the work evaluation unit 104 to evaluate the work of the evaluation worker with high accuracy based on the corrected coordinates of the skeleton of the model worker.
[0025] (Correction example) Fig. 2 is a diagram showing an example of correction of the learning skeleton data 131. Fig. 2 shows an image Ga of a model worker Ha and an image Gb of an evaluation worker Hb. Furthermore, the symbol K roughly indicates a positional deviation (see arrow m) between the camera device 2 when the image Ga is captured and the camera device 2 when the image Gb is captured, as viewed from above the camera device 2.
[0026] The images Ga and Gb are captured by a camera device 2 installed in front of the model worker Ha and evaluation worker Hb in the workplace. In the workplace, for example, a workbench 80 on which work is performed, a parts box 81, and a transport conveyor 83 are installed. For example, the model worker Ha and the evaluation worker Hb take out a part C from the parts box 81 on the right hand side and assemble it on a workpiece W on the workbench 80. The workpiece W is transported by the transport conveyor 83.
[0027] The skeleton detection unit 101 detects the coordinates of the skeletons of the model worker Ha and the evaluation worker Hb from the images Ga and Gb by image analysis using a general-purpose AI. Examples of the skeleton coordinates include the coordinates of the nose position P0, the left shoulder position P23, the right shoulder position P13, the left elbow position P22, the right elbow position P12, the left hand position P21, the right hand position P11, the left hip position P32, and the right hip position P31 of the model worker Ha and the evaluation worker Hb. In the images Ga and Gb, the horizontal direction is the X direction, and the vertical direction is the Y direction.
[0028] Before evaluation, the task evaluation unit 104 has the learning model 133 learn the coordinates of the skeleton in image Ga to generate learned skeleton data 131. During evaluation, the task evaluation unit 104 evaluates the task based on the coordinates of the skeleton in image Gb using the learning model 133 that has already learned the coordinates of the skeleton in image Ga. At this time, the coordinates of the skeleton in image Ga and the coordinates of the skeleton in image Gb are compared with each other, and an evaluation result according to the comparison result is obtained.
[0029] However, as indicated by the symbol K, if the position of the camera device 2 when the image Gb is captured is shifted from the position when the image Ga is captured, the position of the evaluation worker Hb in the image Gb will also be shifted from the position of the model worker Ha in the image Ga. In this example, the camera device 2 has shifted to the left side of the paper surface of FIG. 2, as indicated by the arrow m, so that the evaluation worker Hb is located to the right of the model worker Ha. For this reason, even if the evaluation worker Hb performs the work correctly, the work evaluation unit 104 may output an evaluation result of "fail" due to differences in skeletal coordinates.
[0030] Therefore, the skeleton data correction unit 103 corrects the coordinates of the skeleton of the exemplary worker Ha based on the deviation Δd between the body axis Aa of the exemplary worker Ha and the body axis Ab of the evaluation worker Hb. The position Pm of each skeleton after correction is indicated by a white circle, and is a position obtained by moving the positions P0, P11-P13, P21-P23, P31, and P32 before correction by the deviation Δd between the body axes Aa and Ab in the positive direction of the X axis. Note that in this example, since the evaluation worker Hb is located on the positive side of the X axis from the exemplary worker Ha, the deviation Δd is a positive value, but if the evaluation worker Hb is located on the negative side of the X axis from the exemplary worker Ha, the deviation Δd will be a negative value.
[0031] The axis shift calculation unit 102 calculates the shift Δd between the body axes Aa, Ab from the coordinates of the skeletons of the model worker Ha and the evaluation worker Hb when they are standing in front of the workbench 80. The body axes Aa, Ab pass through the midpoint of the coordinates (P13, P23) of the skeletons of the right and left shoulders and the midpoint of the coordinates (P31, P32) of the right and left hips for the model worker Ha and the evaluation worker Hb. The axis shift calculation unit 102 calculates the difference in the X coordinates of the body axes Aa, Ab as the shift Δd.
[0032] Therefore, it is possible to offset the error in the coordinates of the skeleton caused by the horizontal positional deviation of the camera device 2 by correcting the deviation Δd between the body axes Aa, Ab. Therefore, the task evaluation unit 104 can evaluate the task of the evaluator Hb with high accuracy.
[0033] In addition, since the skeleton data correction unit 103 corrects the skeleton coordinates of the model worker Ha instead of the skeleton coordinates of the evaluation worker Hb, there is no need to correct the skeleton coordinates of the evaluation worker Hb in real time while the evaluation worker Hb is working, and the processing load can be reduced. Note that, although the assembly work is given as an example of the work in this example, the same effect as above can be obtained for other work such as welding work, painting work, and disassembly work.
[0034] (Example of operation of work evaluation server 1) 3 is a flowchart showing an example of the operation of the work evaluation server 1. This operation is executed, for example, when the device control unit 100 receives an evaluation request from the terminal 3 via the LAN 90.
[0035] First, the axis shift calculation unit 102 reads out the coordinates of the skeleton of the model worker during work from the learning skeleton data 131 (step St1). Next, the axis shift calculation unit 102 calculates the X coordinate of the body axis from the coordinates of the skeleton of the model worker (step St2).
[0036] Next, the skeleton detection unit 101 acquires an image of the evaluation worker from the camera device 2 via the LAN 90 (step St3). Next, the skeleton detection unit 101 detects the coordinates of the evaluation worker's skeleton in the image (step St4). Next, the skeleton detection unit 101 judges whether the evaluation worker has started work based on the coordinates of the evaluation worker's skeleton in the image (step St5). At this time, the skeleton detection unit 101 judges whether the evaluation worker is performing an action other than work, such as walking, by comparing the coordinates of the evaluation worker's skeleton with a predetermined pattern.
[0037] If the work has not started (No in step St5), the skeleton coordinates are detected again (step St4). If the work has started (Yes in step St5), the skeleton detection unit 101 stores the skeleton coordinates in time series as detected skeleton data 130 (step St6). Next, the axis deviation calculation unit 102 calculates the X coordinate of the body axis from the skeletal coordinates of the evaluation worker (step St7). Next, the axis deviation calculation unit 102 calculates the difference between the X coordinates of the body axes of the model worker and the evaluation worker as the deviation between the body axes (step St8). Next, the skeleton data correction unit 103 corrects the skeleton coordinates of the learning skeleton data 131 based on the deviation between the body axes (step St9).
[0038] Next, the task evaluation unit 104 compares the skeleton coordinates of the corrected learning skeleton data 131 with the skeleton coordinates of the detected skeleton data 130 (step St10). Next, the task evaluation unit 104 uses the learning model 133 to determine from the comparison result of the skeleton coordinates whether or not the movement of the model worker according to the learning skeleton data 131 and the movement of the evaluation worker according to the detected skeleton data 130 are similar to each other (step St11).
[0039] If the motion of the model worker and the motion of the evaluating worker are similar to each other (Yes in step St11), the task evaluation unit 104 evaluates the task as "good" (step St12), and if the motion of the model worker and the motion of the evaluating worker are not similar to each other (No in step St11), it evaluates the task as "unacceptable" (step St13). Next, the task evaluation unit 104 saves the evaluation result as evaluation data 132 (step St14). The task evaluation server 1 operates in this manner.
[0040] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this embodiment and can be modified in various ways without departing from the scope of the present invention. [Explanation of symbols]
[0041] 1 Task evaluation server (task evaluation device), 10 CPU, 101 skeleton detection unit (detection unit), 102 axis deviation calculation unit (calculation unit), 103 skeleton data correction unit (correction unit), 104 task evaluation unit (evaluation unit), 130 detected skeleton data, 131 learning skeleton data, 133 learning model, Aa, Ab body axis
Claims
[Claim 1] a detection unit that detects coordinates of a skeleton of a first worker in a first image captured by an imaging device and coordinates of a skeleton of a second worker in a second image captured by the imaging device; an evaluation unit that uses a learning model that has been machine-learned on a time series basis of skeletal coordinates of the first worker while performing the work in the first image, and evaluates the work of the second worker based on a time change in skeletal coordinates of the second worker while performing the work in the second image; a calculation unit that calculates a deviation between the body axes of the first worker and the second worker from a skeletal coordinate of the first worker working in the first image and a skeletal coordinate of the second worker working in the second image; a correction unit that corrects the coordinates of the skeleton of the first worker learned by the learning model based on a deviation between the body axes. Work evaluation device.
Citation Information
Patent Citations
Action recognition method, action recognition program and action recognition device
JP2022003434A