Work evaluation device

The work evaluation device effectively identifies and evaluates workers by using skeletal point analysis and threshold-based area calculation, addressing interference and accuracy issues in existing systems.

JP2024158253AActive Publication Date: 2024-11-08TOYOTA JIDOSHA KK
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023073298
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-27
Publication Date
2024-11-08
Estimated Expiration
2043-04-27

AI Technical Summary

Technical Problem

Existing work evaluation systems face challenges in accurately identifying the worker to be evaluated due to the presence of other individuals in the image frame, which can interfere with proper evaluation, and the installation of cameras can obstruct work or hide the worker, leading to incomplete assessments.

Method used

A work evaluation device that includes a detection unit to identify the worker's skeleton, calculates the area of a rectangular region using key skeletal points, and employs threshold values to distinguish the worker to be evaluated, using machine learning to assess work motions based on temporal changes in skeletal coordinates.

Benefits of technology

Enables accurate identification and evaluation of the worker's work by excluding non-evaluable individuals and minimizing camera interference, ensuring precise work assessment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158253000001_ABST
    Figure 2024158253000001_ABST
Patent Text Reader

Abstract

To provide a work evaluation device capable of appropriately evaluating a worker of an evaluation object.SOLUTION: A work evaluation device includes: a detection unit for respectively detecting skeletons of a plurality of persons from an image obtained by imaging the plurality of persons; a calculation unit for calculation the area of a substantially square region having a contour line passing through four points of left and right shoulders and left and right hips in the skeleton of each person; an identification unit for identifying a person whose area is larger than a first threshold among the plurality of persons as a worker of an evaluation object; and an evaluation unit for evaluating work of the worker on the basis of time change of the coordinates of the skeleton of the worker in the image.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an activity evaluation device. [Background technology]

[0002] For example, Patent Document 1 discloses a method for determining whether or not a worker is performing a prescribed movement by extracting skeletal information of the worker from an image capturing the worker's work. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-163293 A Summary of the Invention [Problem to be solved by the invention]

[0004] When evaluating work as described above, the image does not necessarily show only the worker to be evaluated, and depending on the image capture position, other workers other than the worker to be evaluated may be shown in front of or behind the worker, making it difficult to properly evaluate the work. In response to this, it is conceivable to install a camera near the front of the workplace of the worker to be evaluated so that only the worker to be evaluated is shown in the image.

[0005] However, not only is there the risk that the camera will get in the way of work, but there is also the risk that the worker may be hidden in the shadow of a large tool, such as an electric impact wrench hanging from the ceiling of the workplace, making it difficult to properly evaluate the worker's work.

[0006] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a task evaluation device that can appropriately evaluate a worker to be evaluated. [Means for solving the problem]

[0007] The work evaluation device of the present invention includes a detection unit that detects the skeletons of multiple people from an image capturing the multiple people, a calculation unit that calculates the area of ​​an approximately rectangular region having an outline passing through four points on the skeleton of each person, the left and right shoulders and the left and right waists, an identification unit that identifies a person among the multiple people whose area is greater than a first threshold as a worker to be evaluated, and an evaluation unit that evaluates the work of the worker based on changes over time in the coordinates of the skeleton of the worker in the image.

[0008] In the above task evaluation device, the identification unit may identify, from among the plurality of persons, a person whose area is smaller than a second threshold value as a worker to be evaluated.

[0009] In the above-mentioned work evaluation device, the calculation unit may calculate a time average of the area in accordance with a time change in coordinates of the four points of the skeleton of the person in the image, and the identification unit may identify the worker based on the time average of the area.

[0010] In the above-mentioned work evaluation device, the evaluation unit may define a work area in the image where the work is performed based on changes in the coordinates of the worker's skeleton over time, and the detection unit may exclude areas in the image other than the work area from the detection range of the skeleton.

[0011] In the above task evaluation device, the evaluation unit may use a machine-learned discrimination model through supervised learning that receives input of the coordinates of the skeleton in the image and outputs the type of task, and may discriminate and evaluate the type of task based on changes in the coordinates of the skeleton over time. Effect of the Invention

[0012] According to the present invention, it is possible to appropriately evaluate a worker to be evaluated. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram showing an example of a task evaluation system. [Diagram 2] FIG. 2 is a diagram showing an example of an image captured by a camera device for explaining a method for identifying a worker to be evaluated. [Diagram 3] FIG. 3 is a diagram showing an example of an image captured by a camera device of a working motion of a worker. [Figure 4] FIG. 4 is a diagram showing an example of an image that has been subjected to the mask processing. [Diagram 5] FIG. 5 is a flowchart showing an example of the operation of the activity evaluation server. [Figure 6] FIG. 6 is a flowchart showing an example of the operation of evaluating a task. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (Example of the configuration of a work evaluation system) 1 is a configuration diagram showing an example of a work evaluation system 9. As an example, the work evaluation system 9 is installed in an assembly plant for automobiles or the like.

[0015] The task evaluation system 9 includes a task evaluation server 1 and a camera device 2 which are capable of communicating with each other via a LAN (Local Area Network) 90. The task evaluation server 1 is an example of a task evaluation device. The task evaluation server 1 evaluates the task movement of a worker. The camera device 2 captures an image of the worker.

[0016] The work evaluation server 1 is an example of a computer, and has a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, a HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, and the communication port 14 via a bus 19 so as to be able to input and output signals to and from each of them.

[0017] The ROM 11 stores a program that drives the CPU 10. The RAM 12 functions as a working memory for the CPU 10. The communication port 14 is, for example, a wireless LAN (Local Area Network) card, and processes communication of the CPU 10 via a LAN 90.

[0018] When the CPU 10 reads the program from the ROM 11, it forms a device control unit 100, a skeleton detection unit 101, an area calculation unit 102, an evaluation target specification unit 103, and an activity evaluation unit 104 as software functions. In addition, the HDD 13 stores skeleton data 130 and evaluation data 131. The device control unit 100 instructs the skeleton detection unit 101, the area calculation unit 102, the evaluation target specification unit 103, and the activity evaluation unit 104 to operate in accordance with a sequence defined in the program. Note that the device control unit 100, the skeleton detection unit 101, the area calculation unit 102, the evaluation target specification unit 103, and the activity evaluation unit 104 are not limited to software, and may be realized by hardware such as an ASIC (Application Specified Integrated Circuit).

[0019] The skeleton detection unit 101 is an example of a detection unit. The skeleton detection unit 101 detects the skeletons of multiple people from an image of the people captured by the camera device 2. The skeleton detection unit 101 receives images from the camera device 2 via the LAN 90. The skeleton detection unit 101 detects the coordinates of the skeletons of the workers by image analysis using, for example, a general-purpose AI (Artificial Intelligence).

[0020] For example, the skeleton detection unit 101 stores the coordinates of multiple points of each person's skeleton as time-series skeleton data 130 in the HDD 13. The skeleton data 130 indicates, for example, the coordinates of the nose, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip of each person #1 to #N (N: positive integer) in the image for each frame number of the image. The coordinates are indicated by X coordinates and Y coordinates (x, y) when the horizontal and vertical directions of a rectangular frame of the image are defined as the X axis and the Y axis, respectively. There is no limitation on the position of the origin.

[0021] The area calculation unit 102 is an example of a calculation unit. The area calculation unit 102 calculates the area of ​​a substantially rectangular region having an outline passing through four points, the left and right shoulders and the left and right waists, in the skeleton of each person. This region corresponds to the torso of each person. The shape of the torso is, for example, a rectangular shape with corners at the above four points, but is not limited thereto, and may be, for example, a shape of a figure with four rounded corners passing through each of the four points. The camera device 2 is disposed substantially in front of the worker to be evaluated so that the work movement of the worker to be evaluated is captured in the image, so that at least the torso of the worker to be evaluated among the multiple people in the image can be clearly captured. The area calculation unit 102 acquires the coordinates of the four points from the skeleton data 130 and calculates the area by a geometric calculation method. The torso is an example of a substantially rectangular region.

[0022] The evaluation target identification unit 103 is an example of an identification unit. The evaluation target identification unit 103 identifies the worker to be evaluated from multiple people based on the area of ​​the torso. The area of ​​the torso is larger for a person closer to the installation position of the camera device 2 and smaller for a person farther from the installation position of the camera device 2. In other words, the area is determined according to the distance between the camera device 2 and the person. Therefore, the evaluation target identification unit 103 can identify the worker to be evaluated when the area of ​​the torso is within a range according to the distance between the camera device 2 and the worker to be evaluated.

[0023] The evaluation target identification unit 103 identifies a person whose torso area is larger than a threshold value S1 as a worker to be evaluated. Therefore, by appropriately setting the threshold value S1 according to the distance between the camera device 2 and the worker to be evaluated, a person appearing behind the worker to be evaluated in the image from the camera device 2 is excluded from the evaluation target. Note that the threshold value S1 is an example of a first threshold value.

[0024] Furthermore, the evaluation target identification unit 103 identifies a person whose torso area is larger than a threshold value S2 (>S1) as a worker to be evaluated. Therefore, by appropriately setting the threshold value S2 according to the distance between the camera device 2 and the worker to be evaluated, a person who appears in front of the worker to be evaluated in the image from the camera device 2 is excluded from the evaluation target. Note that the threshold value S2 is an example of a second threshold value.

[0025] Therefore, the evaluation target identification unit 103 can easily identify the worker to be evaluated from the area of ​​the torso, which enables the task evaluation unit 104 to appropriately evaluate the task.

[0026] The task evaluation unit 104 is an example of an evaluation unit. The task evaluation unit 104 evaluates the task of a worker based on the change over time in the coordinates of the skeleton of the worker to be evaluated in the image. The task evaluation unit 104 acquires each coordinate of the skeleton of the worker to be evaluated from the skeleton data 130, and determines the motion of the worker from the change over time in each coordinate. For example, the task evaluation unit 104 determines whether the motion of the worker matches a predetermined motion, and records the determination result as evaluation data 131.

[0027] (Example of identifying workers to be evaluated) 2 is a diagram showing an example of an image Ga of the camera device 2 for explaining a method for identifying a worker to be evaluated. The X-axis and Y-axis in the image Ga correspond to the horizontal and vertical directions of the paper, respectively. This also applies to the subsequent drawings.

[0028] The image Ga shows, for example, a work site before work begins. The camera device 2 is installed at a position where, for example, the worker Ha to be evaluated is located approximately in the center of the image Ga, and is sufficiently far away from the workbench 80 of the worker Ha so as not to interfere with the work of the worker Ha. Therefore, the image Ga also shows other workers Hb and people Hc and Hd other than the worker Ha. Examples of the people Hc and Hd include, but are not limited to, visitors and supervisors of the work site. The non-evaluated worker Hb and person Hc are located behind the worker Ha, and the non-evaluated person Hd is located in front of the worker Ha.

[0029] The skeleton detection unit 101 detects the coordinates of the skeletons of the workers Ha, Hb and the persons Hc, Hd from the image Ga. As the coordinates of the skeleton of the worker Ha, for example, the coordinates of the position P0 of the nose of the worker Ha, the position P23 of the left shoulder, the position P13 of the right shoulder, the position P22 of the left elbow, the position P12 of the right elbow, the position P21 of the left hand, the position P11 of the right hand, the position P32 of the left waist, and the position P31 of the right waist can be mentioned. Also, although the symbols are omitted, the coordinates of the skeletons are similarly detected for the other workers Hb and the persons Hc, Hd. Note that the positions P0, P11 to P13, P21 to P23, P31, P32 are shown only in the image Ga, and the illustration is omitted in the images shown later.

[0030] The skeleton detection unit 101 records each coordinate of the skeletons of the workers Ha, Hb and the persons Hc, Hd in the skeleton data 130 respectively. The area calculation unit 102 calculates the areas Sa to Sd of the torso parts of the workers Ha, Hb and the persons Hc, Hd from the skeleton data 130. For example, the area calculation unit 102 calculates the area Sa from the four coordinates of the position P23 of the left shoulder, the position P13 of the right shoulder, the position P32 of the left waist, and the position P31 of the right waist for the skeleton of the worker Ha. Also, the areas Sb to Sd of the torso parts of the other workers Hb and the persons Hc, Hd are calculated in the same way. The area calculation unit 102 compares the areas Sa to Sd with predetermined threshold values S1, S2.

[0031] The threshold values S1, S2 define a range including the area Sa of the torso part of the worker Ha to be evaluated, for example, based on the distance between the camera device 2 and the worker Ha to be evaluated. For this reason, for the area Sa, S1 < Sa < S2 holds, but the areas Sb, Sc are less than or equal to the threshold value S1, and the area Sd is greater than or equal to the threshold value S2. Therefore, the evaluation target specifying unit 103 specifies the worker Ha to be evaluated by determining that S1 < Sa < S2 holds.

[0032] Furthermore, the area calculation unit 102 may calculate the time average of the areas Sa to Sd according to the time change of the coordinates in the skeleton image Ga. In this case, the evaluation target specification unit 103 determines whether or not the time average of the areas Sa to Sd is within the range of the thresholds S1 and S2, as described above. This makes it possible to reduce the influence of the time change of the positions and orientations of the workers Ha, Hb and the persons Hc, Hd on the calculation of the areas Sa to Sd.

[0033] (Example of work operations) Fig. 3 is a diagram showing an example of images G1b, G2b captured by the camera device 2 of the working movement of a worker Ha. In Fig. 3, the same components as those in Fig. 2 are given the same reference numerals, and the description thereof will be omitted.

[0034] In this example, the work operation using the part C includes, but is not limited to, an operation of assembling the part C acquired in the acquisition operation in Fig. 2 to the work W. Examples of the work operation include an operation of painting the part C, an operation of welding the part C to the work W, and an operation of inspecting the part C.

[0035] The work evaluation unit 104 identifies the work motion of the worker Ha based on the change over time in the coordinates of each part of the skeleton of the worker Ha identified by the evaluation target identification unit 103. Image G1b shows the worker Ha holding the workpiece W with his left hand and aligning the installation position of the part C with his right hand, and image G2b shows the worker Ha assembling the part C to the workpiece W. The work evaluation unit 104 determines the work motion by pattern matching of the change over time in the coordinates of the skeleton in images G1b and G2b and the coordinates of the skeleton in images of other frames not shown. The work evaluation unit 104 evaluates whether the work motion is a specified motion or not by pattern matching, and generates evaluation data 131.

[0036] The task evaluation unit 104 distinguishes various task actions in addition to those described above. For example, AI can be used for this distinction. For example, the task evaluation unit 104 distinguishes and evaluates the type of task using a machine-learned discrimination model based on supervised learning that receives the coordinates of the skeleton in the images G1b and G2b as input and outputs the type of task action.

[0037] Specifically, in the learning process of the discrimination model, annotation is performed on the type of worker's movement based on the time change of the coordinates of the skeleton of the teacher data. The discrimination model is a neural network that mathematically models the function of the human brain, and the weight coefficient of the activation function of the part corresponding to the neuron is determined based on machine learning. In this way, the task evaluation unit 104 can discriminate task movements with high accuracy by using a discrimination model that has been machine-learned by supervised learning.

[0038] (Image masking example) Fig. 4 is a diagram showing an example of an image Gc that has been subjected to mask processing. In Fig. 4, the same components as those in Fig. 2 are given the same reference numerals, and the description thereof will be omitted.

[0039] The task evaluation unit 104 defines a task area R in the image Gc where the task is performed based on the time-dependent changes in the coordinates of the skeleton of the worker Ha. For example, when the task evaluation unit 104 determines a specific task movement, it defines the task area R based on the range in which the coordinates of each part of the skeleton are displaced during the task movement. The task evaluation unit 104 masks the areas in the image Gc other than the task area R, as shown by shading.

[0040] The skeleton detection unit 101 excludes areas in the image Gc other than the working area R from the skeleton detection range. In other words, the skeleton detection unit 101 does not detect skeletons from areas in the image Gc other than the working area R. This allows the skeleton detection unit 101 to reduce the load of detecting the skeletons of the worker Hb and persons Hc and Hd who are not the evaluation targets.

[0041] (Example of the operation of the work evaluation server) 5 is a flowchart showing an example of the operation of the task evaluation server 1. First, the skeleton detection unit 101 receives image data from the camera device 2 via the LAN 90 (step St1). Next, the skeleton detection unit 101 detects the skeletons of each of the multiple persons #1 to #N from the image of the image data (step St2). Next, the skeleton detection unit 101 records the skeleton data 130 for each of the persons #1 to #N (step St3).

[0042] Next, the area calculation unit 102 selects a person #i (i: an integer from 1 to N) (step St4). Next, the area calculation unit 102 calculates the area S from the four coordinates of the position P23 of the left shoulder, the position P13 of the right shoulder, the position P32 of the left waist, and the position P31 of the right waist of the person #i being selected from the skeleton data 130 (step St5).

[0043] Next, the evaluation target identification unit 103 compares the area S with the threshold values S1 and S2 respectively (step St6). When S1 < S < S2 holds (Yes in step St6), the evaluation target identification unit 103 sets the person #i being selected as the operator to be evaluated (step St7). Next, the work evaluation unit 104 evaluates the operator to be evaluated (step St8). Note that an example of the evaluation operation will be described later. After that, this operation ends.

[0044] Also, when S1 < S < S2 does not hold (No in step St6), the area calculation unit 102 determines the presence or absence of unselected persons #1 to #N (step St9). When there are unselected persons #1 to #N (Yes in step St9), the operations after step St4 are performed again. At this time, the area calculation unit 102 selects another person #i. Also, when there are no unselected persons #1 to #N (No in step St9), this operation ends.

[0045] Note that in step St6 above, the area S is compared with the threshold values S1 and S2. However, when there is no other person in front of the operator to be evaluated in the image of the camera device 2, the area S may be compared with only the threshold value S1. In this case, when S1 < S holds, the operator to be evaluated is identified.

[0046] FIG. 6 is a flowchart showing an example of the work evaluation operation. This operation is executed in step St8 of FIG. 7.

[0047] First, the skeleton detection unit 101 receives image data from the camera device 2 via the LAN 90 (step St11). Next, the skeleton detection unit 101 detects the skeletons of one or more persons #1 to #N from the image of the image data (step St12). Next, the skeleton detection unit 101 records the skeleton data 130 of the persons #1 to #N (step St13).

[0048] Next, the task evaluation unit 104 compares the displacement of the skeletal coordinates of the worker to be evaluated within a predetermined period with a threshold value TH (step St14). For example, the task evaluation unit 104 calculates the displacement of the skeletal coordinates for a predetermined number of frames. If the displacement of the skeletal coordinates of the worker to be evaluated within a predetermined period is greater than the threshold value TH (Yes in step St14), the evaluation target identification unit 103 determines that the worker to be evaluated has left the workplace and performs the identification process again (step St20). At this time, the evaluation target identification unit 103 identifies the worker again by the method shown in FIG. 5. If the re-identification is completed (Yes in step St21), the operation of step St19 described later is performed, and if the re-identification is not completed (No in step St21), the operation of step St20 is performed again.

[0049] If the displacement of the skeletal coordinates of the worker to be evaluated within the predetermined period is equal to or less than the threshold value TH (No in step St14), the evaluation target specification unit 103 specifies the work motion of the worker based on the time change of the skeletal coordinates of the worker in the image (step St15). Next, the work evaluation unit 104 evaluates the work by comparing the specified work motion with a specified motion (step St16). At this time, the work evaluation unit 104 records the evaluation result in the evaluation data 131.

[0050] Next, if the working area R in the image has not been defined (No in step St17), the work evaluation unit 104 defines the working area R from the time change in the coordinates of the worker's front and performs mask processing on the area other than the working area R (step St18). The skeleton in the masked area is no longer detected in step St12. Also, if the working area R in the image has been defined (Yes in step St17), step St18 is not performed.

[0051] Next, the task evaluation unit 104 judges whether all tasks have been completed based on the skeleton data 130 (step St19). If there is any unfinished task (No in step St19), the processes from step St11 onward are performed again, and if all tasks have been completed (Yes in step St19), this process ends. In this manner, the task evaluation server 1 operates.

[0052] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this embodiment and can be modified in various ways without departing from the scope of the present invention. [Explanation of symbols]

[0053] 1 Task evaluation server (task evaluation device), 10 CPU, 101 skeleton detection unit (detection unit), 102 area calculation unit (calculation unit), 103 evaluation target specification unit (specification unit), 104 task evaluation unit (evaluation unit), 130 skeleton data, Ha, Hb worker, Hc, Hd person

Claims

1. A detection unit that detects skeletons of a plurality of people from an image capturing the plurality of people; a calculation unit that calculates an area of ​​a substantially rectangular region having an outline passing through four points, i.e., the left and right shoulders and the left and right waists, of the skeleton of each person; an identification unit that identifies, among the plurality of persons, a person whose area is greater than a first threshold value as a worker to be evaluated; an evaluation unit that evaluates the work of the worker based on a time change in coordinates of the skeleton of the worker in the image, Work evaluation device.

2. The identification unit identifies, among the plurality of persons, a person whose area is smaller than a second threshold value as a worker to be evaluated. The task evaluation device according to claim 1 .

3. the calculation unit calculates a time average of the area in accordance with a time change in coordinates of the four points of the skeleton in the image for each person; The identification unit identifies the worker based on a time average of the area. The task evaluation device according to claim 1 or 2.

4. The evaluation unit defines a work area in which the work is performed in the image based on a time change in coordinates of the skeleton of the worker, The detection unit excludes an area other than the working area in the image from the detection range of the skeleton. The task evaluation device according to claim 1 or 2.

5. the evaluation unit uses a machine-learned discrimination model through supervised learning, which receives coordinates of the skeleton in the image as input and outputs the type of the task, and discriminates and evaluates the type of the task based on a change in the coordinates of the skeleton over time. The task evaluation device according to claim 1 or 2.

Citation Information

Patent Citations

  • Operation specification device, operation specification method, and operation specification program

    JP2020198133A

  • Skeleton detection system and work management device

    WO2021250931A1

  • Work analyzer and work analysis program

    JP2021163293A