Work evaluation device

Abstract

To provide a work evaluation device capable of properly evaluating the work of a worker.SOLUTION: A work evaluation server as a work evaluation device is configured to operate as a detection unit that, when receiving an evaluation request from a terminal, acquires images of a worker at work from a camera device and detects multiple skeleton coordinates of the working worker in the image in chronological order; and an evaluation unit that evaluates the work based on the time change of the coordinates of the multiple skeletons by using the importance assigned to each skeleton according to the type of the work.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an activity evaluation device. [Background technology]

[0002] For example, Patent Document 1 describes a service that recognizes human behavior from camera footage based on skeletal information, using a learning model for object recognition in which basic movements have been learned according to machine learning algorithms such as deep learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-3434 Summary of the Invention [Problem to be solved by the invention]

[0004] When the skeletal coordinates of a person working are detected from a video image and used for evaluation as described above, there is a risk that the accuracy of the evaluation process will be insufficient and the work will not be evaluated appropriately.

[0005] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide an activity evaluation device that can appropriately evaluate activities. [Means for solving the problem]

[0006] The work evaluation device of the present invention has a detection unit that detects the coordinates of multiple skeletal structures of a worker working in an image in chronological order, and an evaluation unit that uses an importance assigned to each of the skeletons to evaluate the work based on changes in the coordinates of the multiple skeletal structures over time.

[0007] In the above-mentioned work evaluation device, the evaluation unit may calculate, for each skeleton, a distance between the coordinates of multiple skeletons of another worker performing the work detected in advance from another image and the coordinates of the multiple skeletons, and evaluate the work by calculating a product-sum of the distance and the importance for each skeleton in chronological order.

[0008] Another work evaluation device of the present invention includes a detection unit that detects skeletal coordinates of a first worker performing a task in an image in accordance with a time series, an evaluation unit that uses a learning model that has been machine-learned to learn the skeletal coordinates of a second worker performing the task in another image in accordance with the time series, and evaluates the task by comparing the skeletal coordinates of the first worker with the skeletal coordinates of the second worker in accordance with the time series, and a data processing unit that adds or compresses data of the skeletal coordinates of one of the first worker and the second worker so as to reduce the difference in the number of data points over the time series of the skeletal coordinates of the first worker and the second worker used in evaluating the task.

[0009] In the above task evaluation device, the data processing unit may add the data by performing an interpolation process on data of coordinates of a skeleton of one of the first worker and the second worker.

[0010] In the above task evaluation device, the data processing unit may compress the data by performing an averaging process on data of coordinates of a skeleton of one of the first worker and the second worker. Effect of the Invention

[0011] According to the present invention, work can be appropriately evaluated. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram showing an example of a task evaluation system. [Diagram 2] FIG. 2 is a plan view showing an example of an image captured of an evaluation worker during work. [Diagram 3]FIG. 3 is a diagram illustrating an example of the importance table. [Figure 4] FIG. 4 is a diagram showing an example of adding and compressing data of skeleton coordinates. [Diagram 5] FIG. 5 is a flowchart showing an example of the operation of the activity evaluation server. [Figure 6] FIG. 6 is a flowchart illustrating an example of the data number adjustment process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (Example of the configuration of a work evaluation system) 1 is a configuration diagram showing an example of a work evaluation system 9. As an example, the work evaluation system 9 is installed in an assembly plant for automobiles or the like.

[0014] The work evaluation system 9 includes a work evaluation server 1, a camera device 2, and a terminal 3 which are capable of communicating with each other via a LAN (Local Area Network) 90. The work evaluation server 1 is an example of a work evaluation device. The work evaluation server 1 evaluates the work actions of a worker. The camera device 2 is an example of an imaging device, and captures images of people such as workers. The terminal 3 is carried, for example, by the worker or a related person, and receives work evaluation results from the work evaluation server 1.

[0015] The work evaluation server 1 has a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, a HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, and the communication port 14 via a bus 19 so as to be able to input and output signals among them.

[0016] When the CPU 10 reads a program from the ROM 11, it forms, as software functions, a device control unit 100, a skeleton detection unit 101, a data processing unit 102, and an operation evaluation unit 103. In addition, the HDD 13 stores detected skeleton data 130, learned skeleton data 131, evaluation data 132, a learning model 133, and an importance table 134.

[0017] The device control unit 100 instructs the skeleton detection unit 101, data processing unit 102, and task evaluation unit 103 to operate in accordance with a sequence defined in a program. Note that the device control unit 100, data processing unit 102, and task evaluation unit 103 are not limited to being realized by software, and may be realized by hardware circuits such as an ASIC (Application Specified Integrated Circuit).

[0018] The skeleton detection unit 101 is an example of a detection unit. The skeleton detection unit 101 detects the coordinates of the skeleton of a person in an image captured by the camera device 2. The skeleton detection unit 101 receives image data from the camera device 2 via the LAN 90. The skeleton detection unit 101 detects the coordinates of the skeleton of the person by image analysis using, for example, a general-purpose AI (Artificial Intelligence).

[0019] When the task evaluation unit 103 evaluates a worker to be evaluated (hereinafter referred to as the evaluation worker), the camera device 2 captures an image of the evaluation worker. In addition, the camera device 2 captures an image of an exemplary worker (hereinafter referred to as the exemplary worker) prior to the evaluation of the evaluation worker, in order to allow the learning model 133 used by the task evaluation unit 103 for evaluation to learn the coordinates of a skeleton of an evaluation standard. Note that the evaluation worker is an example of a first worker, and the exemplary worker is an example of a second worker.

[0020] The skeleton detection unit 101 detects the coordinates of the evaluation worker's skeleton from the image of the evaluation worker and stores them in the HDD 13 as detected skeleton data 130. The detected skeleton data 130 indicates, for example, the coordinates of the evaluation worker's face, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip in the image in chronological order for each frame number of the image. The coordinates are indicated by the X coordinate and the Y coordinate (x, y) when the horizontal and vertical directions of the rectangular frame of the image are defined as the X axis and the Y axis, respectively. There is no limitation on the position of the origin.

[0021] 2 is a plan view showing an example of an image Ga captured of an evaluation worker Ha during work. The image Ga is captured by a camera device 2 installed in front of the evaluation worker Ha in the workplace. In the workplace, as an example, a workbench 80 on which work is performed, a parts box 81, a transport conveyor 83, and a press machine 82 are installed. As an example, the evaluation worker Ha takes out a part C from the parts box 81 on the left hand side and assembles it on a workpiece W on the workbench 80. The workpiece W is transported by the transport conveyor 83.

[0022] The skeleton detection unit 101 detects the coordinates of the skeleton of the evaluator Ha from the image Ga by image analysis using a general-purpose AI. Examples of the skeleton coordinates include the coordinates of the face position P0 of the evaluator Ha, the left shoulder position P23, the right shoulder position P13, the left elbow position P22, the right elbow position P12, the left hand position P21, the right hand position P11, the left hip position P32, and the right hip position P31. In the image Ga, the horizontal direction is the X direction, and the vertical direction is the Y direction.

[0023] 1 again, the task evaluation unit 103 is an example of an evaluation unit. The task evaluation unit 103 uses a learning model 133 that is machine-learned in time series on the coordinates of the skeleton of an exemplary worker working in an image other than the one of the worker to be evaluated, and evaluates the task by comparing the coordinates of the skeleton of the exemplary worker in time series with the coordinates of the skeleton of the exemplary worker. Here, the skeleton refers to, for example, the face, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip.

[0024] The learning model 133 is stored in the HDD 13 as a set of various calculation modules and parameters. In the learning process of the learning model 133, annotation is performed on the type of work using the time change in the coordinates of the skeleton of a model worker during work as teacher data. The learning model 133 is a neural network that mathematically models the functions of the human brain, and determines the weight coefficients of the activation functions of the parts corresponding to neurons based on machine learning. In this way, the work evaluation unit 103 can output evaluation results with high accuracy by using the learning model 133 that has been machine-learned through supervised learning.

[0025] The coordinates of the skeleton of the model worker learned by the learning model 133 are stored in the HDD 13 as learned skeleton data 131. Like the detected skeleton data 130, the learned skeleton data 131 indicates the coordinates of the face, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip of the evaluated worker in the image in chronological order for each frame number of the image. The learning model 133 determines the type of task from the change over time in the detected skeleton data 130, and outputs the learned skeleton data 131 of the corresponding task to the task evaluation unit 103.

[0026] The task evaluation unit 103 evaluates the task of the evaluator based on the time-dependent change in the coordinates of each skeleton, using the importance assigned to each skeleton. The importance indicates the degree of influence that the skeleton coordinates have on the evaluation. The task evaluation unit 103 obtains the importance according to the type of task recognized by the learning model 133 from the importance table 134.

[0027] FIG. 3 is a diagram showing an example of the importance table 134. In the importance table 134, evaluation items, types of work, and the importance corresponding to each of the above skeletons are registered in advance. Evaluation items are points of focus in evaluating a work, and in this example, include, but are not limited to, accuracy, abnormality, proficiency, and load. Accuracy is the accuracy of the work, abnormality is abnormality in the work cycle, proficiency is the sophistication of the work, and load is the physical load of the evaluating worker. In addition, types of work include, but are not limited to, part removal, assembly, and completion confirmation.

[0028] The importance is determined for each skeleton according to the evaluation item and the type of work. For example, when evaluating accuracy, the importance of the left hand (1.0) is greater than the importance of the right hand (0.2) on the premise that the evaluating worker picks up a part with his / her left hand in the part-picking work. When evaluating proficiency, the importance of the right hand (0.4) when evaluating proficiency is greater than the importance of the right hand (0.2) when evaluating accuracy on the premise that the position of the right hand in the part-picking work is more important than that in the case of evaluating accuracy. Note that the importance may be determined arbitrarily by the administrator of the work evaluation server 1 based on the viewpoint of evaluation, for example, or may be determined using the contribution of each skeleton in the machine learning of the learning model 133.

[0029] The task evaluation unit 103 evaluates the task based on the time change of the coordinates of the multiple skeletons of the evaluator, using the importance assigned to each skeleton. Therefore, the task evaluation server 1 can improve the accuracy of the evaluation according to the viewpoint, etc., and appropriately evaluate the task, compared to the case where the coordinates of multiple skeletons are used in the evaluation with equal weighting.

[0030] For example, the task evaluation unit 103 calculates the distance between the coordinates of multiple skeletons of the model worker and the coordinates of multiple skeletons of the evaluated worker for each skeleton, and calculates the sum of the products of the distance and importance for each skeleton in chronological order.

[0031] f(n) = ΣKi × {(Xai - Xbi) 2 +(Yai-Ybi) 2} 1 / 2 (1) S = Σf(n) (2)

[0032] The activity evaluation unit 103 calculates the activity evaluation score S, for example, from the above formulas (1) and (2). In this example, the smaller the evaluation score S, the better the evaluation.

[0033] Equation (1) calculates the evaluation score f(n) (n:1,2,3,...) of the nth frame in the detected skeleton data 130 from the coordinates (Xai,Yai) of the i-th (i:1,2,3,...) skeleton of the evaluated worker in the detected skeleton data 130, the coordinates (Xbi,Ybi) of the i-th (i:1,2,3,...) skeleton of the model worker in the learned skeleton data 131, and the importance Ki corresponding to the i-th skeleton. Furthermore, "Σ" in equation (1) represents the sum of the values ​​in {} corresponding to each of i=1 to j, where j (a positive integer) is the number of skeletons for each frame in the detected skeleton data 130 (j=9 in this example). That is, equation (1) represents the sum of the products of the distance between the coordinates (Xai,Yai) of the skeleton (i) of the evaluated worker and the coordinates of the skeleton (i) of the model worker, and the importance Ki corresponding to the skeleton.

[0034] Equation (2) is the evaluation score S obtained by summing up the evaluation scores f(n) for each frame for the entire task. "Σ" in equation (2) represents the sum of the evaluation scores f(n) for each frame of frame numbers n=1 to m (m: positive integer) of the images capturing the task. For example, when the task evaluation unit 103 evaluates the task of removing parts, when frame numbers "1" to "30" (i.e., m=30) of the detected skeleton data 130 and the learned skeleton data 131 correspond to images of the task of removing parts, the evaluation score S is the sum of the evaluation scores f(1) to f(30) for each frame. Note that the task evaluation unit 103 may average the evaluation score S by the number of frames m.

[0035] In this way, the task evaluation unit 103 calculates the sum of the products of the distances between the coordinates of each skeleton and the importance in chronological order, so that the task evaluation unit 103 can evaluate the task with high accuracy by reflecting the importance in the deviations between the coordinates of each skeleton.

[0036] The data processing unit 102 adds or compresses data of the skeletal coordinates of one of the evaluator and model worker so as to reduce the difference in the number of data points on the time series of the skeletal coordinates of the evaluator and model worker used in the task evaluation. This allows the task evaluation unit 103 to fully compare the skeletal coordinates of the evaluator and model worker, and therefore improves the accuracy of the evaluation and allows the task to be appropriately evaluated, compared to a case where there is a large difference in the number of data points on the time series of the skeletal coordinates of each.

[0037] 4 is a diagram showing an example of adding and compressing data of the coordinates of the skeleton. In this example, the data processing unit 102 adds and compresses data of the X coordinate of the left elbow of the detected skeleton data 130, but the addition and compression of other skeletons and the learning skeleton data 131 are also performed in the same manner as described below.

[0038] Symbol Ta is a part of the detected skeleton data 130, and is an example of data of the X coordinate of the left elbow corresponding to frame numbers "1" to "6" of the detected skeleton data 130. Note that the Y coordinate of the left elbow is omitted, but its addition and compression are similar to the X coordinate.

[0039] Symbol Tb is an example in which data of the X coordinate of the left elbow in frame numbers "1+" to "5+" is added to the data of symbol Ta. The data processing unit 102 adds data by performing linear interpolation processing on the data of the X coordinate of the left elbow in frame numbers "1" to "6".

[0040] The data processing unit 102 calculates the X-coordinate data of the left elbow for frame numbers "1+" to "5+" by linear interpolation processing. For example, the X-coordinate (101) for frame number "1+" is the average value of the X-coordinates (100, 102) for frame numbers "1" and "2," and the X-coordinate (103) for frame number "2+" is the average value of the X-coordinates (102, 104) for frame numbers "2" and "3."

[0041] By using linear interpolation in this way, the data processing unit 102 can add the data while maintaining the accuracy of the coordinates of the left elbow. Note that linear interpolation is an example of an interpolation process, and an interpolation process other than linear interpolation may be used.

[0042] Moreover, code Tc is an example in which the data of code Ta is compressed to data of the X-coordinate of the left elbow in frame numbers "1-" to "3-". The data processing unit 102 compresses the data by performing an averaging process on the data of the X-coordinate of the left elbow in frame numbers "1" to "6". For example, the X-coordinate (101) of frame number "1-" is the average value of the X-coordinates (100, 102) of frame numbers "1" and "2", and the X-coordinate (101) of frame number "2-" is the average value of the X-coordinates (104, 98) of frame numbers "3" and "4".

[0043] By using linear interpolation in this manner, the data processor 102 can compress the data while maintaining the accuracy of the coordinates of the left elbow.

[0044] (Work evaluation server operation) 5 is a flowchart showing an example of the operation of the work evaluation server 1. This operation is executed when the device control unit 100 receives an evaluation request from the terminal 3 via the LAN 90, for example.

[0045] First, the device control unit 100 judges whether the evaluation worker has started work (step St1). At this time, the device control unit 100 may judge the start of work by, for example, analyzing an image of the evaluation worker captured by the camera device 2 using the learning model 133. If the work has not started (No in step St1), the process of step St1 is executed again.

[0046] Furthermore, when the work is started (Yes in step St1), the skeleton detection unit 101 acquires an image of the evaluation worker from the camera device 2 via the LAN 90 (step St2). Next, the skeleton detection unit 101 detects the coordinates of the skeleton of the evaluation worker in the image (step St3) and stores it as detected skeleton data 130 (step St4).

[0047] First, the device control unit 100 judges whether the evaluation worker has finished the work (step St5). At this time, the device control unit 100 may judge the end of the work by analyzing the image of the evaluation worker captured by the camera device 2 using the learning model 133. If the work has not been finished (No in step St5), the operations of steps St2 to St4 are executed again.

[0048] If the task is completed (Yes in step St5), the task evaluation unit 103 reads out the learned skeleton data 131 according to the type of task performed by the evaluating worker (step St6). At this time, for example, the task evaluation unit 103 recognizes the type of task from the detected skeleton data 130 using the learned model 133.

[0049] Next, the data processing unit 102 performs an adjustment process of the difference in the number of pieces of data in the detected skeleton data 130 and the learned skeleton data 131 according to the type of work, as described with reference to Fig. 4 (step St7). The details of this process will be described later.

[0050] Next, the task evaluation unit 103 reads out the importance of each skeleton according to the task type from the importance table 134 (step St8). Next, the task evaluation unit 103 calculates the evaluation score S, for example, from the above formulas (1) and (2) (step St9), and stores it as evaluation data 132 (step St10). Note that the task evaluation unit 103 may transmit the task evaluation score S to the terminal 3 upon request. In this manner, the task evaluation server 1 operates.

[0051] 6 is a flowchart showing an example of the data amount adjustment process, which is executed in step St7 described above.

[0052] First, the data processing unit 102 judges whether the number of data items in the learned skeleton data 131 is greater than the number of data items in the detected skeleton data 130 (step St70). If the number of data items in the learned skeleton data 131 is greater than the number of data items in the detected skeleton data 130 (Yes in step St70), the data processing unit 102 executes linear interpolation processing on the detected skeleton data 130 (step St71). This increases the number of data items in the detected skeleton data 130 to be equal to or close to the number of data items in the learned skeleton data 131, thereby improving the accuracy of comparison of the coordinates of each skeleton.

[0053] If the number of data items in the learned skeleton data 131 is not greater than the number of data items in the detected skeleton data 130 (No in step St70), the data processing unit 102 judges whether the number of data items in the learned skeleton data 131 is less than the number of data items in the detected skeleton data 130 (step St72). If the number of data items in the learned skeleton data 131 is less than the number of data items in the detected skeleton data 130 (Yes in step St72), the data processing unit 102 performs an averaging process on the detected skeleton data 130 (step St73). This reduces the number of data items in the detected skeleton data 130 to the same number as or close to the number of data items in the learned skeleton data 131, improving the accuracy of comparison of the coordinates of each skeleton.

[0054] If the number of data items in the learning skeleton data 131 is the same as the number of data items in the detected skeleton data 130 (No in step St72), the data processing unit 102 does not execute any processing. In this manner, the data number adjustment processing is executed. Note that, although this example shows an example in which the number of data items in the detected skeleton data 130 is increased or decreased, it is also possible to increase or decrease the number of data items in the learning skeleton data 131 in the same manner as above.

[0055] In addition, in this example, the work evaluation unit 103 compares the coordinates of each skeleton in the detected skeleton data 130 and the learned skeleton data 131, but is not limited to this. For example, in order to evaluate the safety of the work of the evaluation worker, the work evaluation unit 103 may compare the coordinates of each skeleton in the detected skeleton data 130 with, for example, the coordinates of a press machine in the image Ga in FIG. 2.

[0056] In this case, the work evaluation unit 103 calculates the evaluation score f(n) by, for example, replacing (Xbi, Ybi) in the above formula (1) with the coordinates of the press machine 82. That is, the work evaluation unit 103 calculates the sum of the distance between the skeleton of the evaluator Ha and the press machine 82 and the product of the importance of each skeleton in chronological order. Since the right hand of the evaluator Ha is close to the press machine 82 in the image Ga, for example, the importance of the right hand may be set higher than the importance of the other skeletons. Note that in this example, the press machine 82 is given as the target of the distance calculation, but the present invention is not limited to this, and other machines may be the target of the distance calculation.

[0057] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this embodiment and can be modified in various ways without departing from the scope of the present invention. [Explanation of symbols]

[0058] 1 Task evaluation server (task evaluation device), 10 CPU, 101 skeleton detection unit (detection unit), 102 data processing unit, 103 task evaluation unit (evaluation unit), 130 detected skeleton data, 131 learning skeleton data, 133 learning model, 134 importance table

Claims

1. a detection unit that detects coordinates of a plurality of skeletons of a worker working in an image in chronological order; and an evaluation unit that evaluates the work based on a time change in coordinates of the plurality of skeletons by using the importance assigned to each skeleton. Work evaluation device.

2. the evaluation unit calculates, for each skeleton, a distance between a coordinate of a plurality of skeletons of the other worker performing the work detected in advance from another image and the coordinate of the plurality of skeletons, and evaluates the work by calculating a product sum of the distance and the importance for each skeleton in chronological order. The task evaluation device according to claim 1 .

3. a detection unit that detects chronological coordinates of a skeleton of a first worker in an image during a task; an evaluation unit that evaluates the work by comparing the skeletal coordinates of the first worker with the skeletal coordinates of the second worker in a time series using a learning model that has been machine-learned based on the skeletal coordinates of the second worker performing the work in another image in a time series; a data processing unit that adds or compresses data of skeletal coordinates of one of the first worker and the second worker so as to reduce a difference in the number of time-series data of the skeletal coordinates of each of the first worker and the second worker used in the evaluation of the work. Work evaluation device.

4. the data processing unit adds the data by performing an interpolation process on data of coordinates of a skeleton of one of the first worker and the second worker. The task evaluation device according to claim 3 .

5. the data processing unit compresses the data by performing an averaging process on data of coordinates of the skeleton of one of the first worker and the second worker. The task evaluation device according to claim 3 or 4.

Images