Work evaluation device

Abstract

To provide a work evaluation device capable of appropriately evaluating a worker of an evaluation object.SOLUTION: A work evaluation device includes: a detection unit for respectively detecting skeletons of a plurality of persons from an image obtained by imaging the plurality of persons; a calculation unit for calculation the area of a substantially square region having a contour line passing through four points of left and right shoulders and left and right hips in the skeleton of each person; an identification unit for identifying a person whose area is larger than a first threshold among the plurality of persons as a worker of an evaluation object; and an evaluation unit for evaluating work of the worker on the basis of time change of the coordinates of the skeleton of the worker in the image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a task evaluation device. [Background technology]

[0002] For example, Patent Document 1 discloses that whether or not a worker is performing a prescribed movement is determined by extracting skeletal information of the worker from an image of the worker's work. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-163293 Summary of the Invention [Problem to be solved by the invention]

[0004] When evaluating work as described above, the image does not necessarily capture only the worker being evaluated, and depending on the image capture position, other workers other than the worker being evaluated may be captured in front of or behind the worker, making it difficult to properly evaluate the work. To address this, it is possible to install a camera near the front of the workplace of the worker being evaluated so that only the worker being evaluated is captured in the image.

[0005] However, not only is there a risk that the camera will get in the way of work, but there is also a risk that the worker may be hidden in the shadow of large tools such as an electric impact hammer hanging from the ceiling of the workplace, making it impossible to properly evaluate the worker's work.

[0006] The present invention has been made in view of the above-mentioned problems, and has an object to provide a task evaluation device that can appropriately evaluate a worker to be evaluated. [Means for solving the problem]

[0007] The task evaluation device of the present invention includes a detection unit that detects the skeletons of a plurality of people from an image of the plurality of people, and a detection unit that detects four points of the skeletons of each person, i.e., the left and right shoulders and the left and right waists. According to the time change of the coordinates in the image, The area of ​​a roughly rectangular region having an outline passing through Time average of a calculation unit for calculating the area of ​​the plurality of people; Time average of The apparatus includes an identification unit that identifies a person whose skeletal coordinates are greater than a first threshold as a worker to be evaluated, and an evaluation unit that evaluates the work of the worker based on changes over time in the coordinates of the skeleton of the worker in the image.

[0008] In the above task evaluation device, the identification unit is configured to identify the person among the plurality of people Time average of A person whose σ is smaller than the second threshold may be identified as the worker to be evaluated.

[0010] In the above-mentioned work evaluation device, the evaluation unit may define a work area in the image where the work is performed based on changes in the coordinates of the worker's skeleton over time, and the detection unit may exclude areas in the image other than the work area from the detection range of the skeleton.

[0011] In the above task evaluation device, the evaluation unit may use a machine-learned discrimination model through supervised learning that receives input of the coordinates of the skeleton in the image and outputs the type of task, and may determine and evaluate the type of task based on changes in the coordinates of the skeleton over time. [Effects of the Invention]

[0012] According to the present invention, it is possible to appropriately evaluate a worker to be evaluated. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram showing an example of a task evaluation system. [Figure 2] FIG. 2 is a diagram showing an example of an image taken by a camera device to explain a method for identifying a worker to be evaluated. [Figure 3] FIG. 3 is a diagram showing an example of an image captured by a camera device showing the work movements of a worker. [Figure 4] FIG. 4 is a diagram showing an example of an image subjected to mask processing. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the task evaluation server. [Figure 6] FIG. 6 is a flowchart showing an example of the operation for evaluating a task. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Example of work evaluation system configuration) 1 is a configuration diagram showing an example of a work evaluation system 9. As an example, the work evaluation system 9 is installed in an assembly plant for automobiles or the like.

[0015] The work evaluation system 9 includes a work evaluation server 1 and a camera device 2 that can communicate with each other via a LAN (Local Area Network) 90. The work evaluation server 1 is an example of a work evaluation device. The work evaluation server 1 evaluates the work movements of a worker. The camera device 2 captures an image of the worker.

[0016] The task evaluation server 1 is an example of a computer, and includes a CPU (Central Processing Unit) 10, a ROM (Read Only Memory) 11, a RAM (Random Access Memory) 12, a HDD (Hard Disk Drive) 13, and a communication port 14. The CPU 10 is electrically connected to the ROM 11, the RAM 12, the HDD 13, and the communication port 14 via a bus 19 so as to be able to input and output signals from and to each other.

[0017] The ROM 11 stores a program that drives the CPU 10. The RAM 12 functions as a working memory for the CPU 10. The communication port 14 is, for example, a wireless LAN (Local Area Network) card, and processes communication of the CPU 10 via the LAN 90.

[0018] When the CPU 10 reads a program from the ROM 11, it forms, as software functions, a device control unit 100, a skeleton detection unit 101, an area calculation unit 102, an evaluation target identification unit 103, and an activity evaluation unit 104. In addition, the HDD 13 stores skeleton data 130 and evaluation data 131. The device control unit 100 instructs the skeleton detection unit 101, the area calculation unit 102, the evaluation target identification unit 103, and the activity evaluation unit 104 to operate in accordance with a sequence defined in the program. Note that the device control unit 100, the skeleton detection unit 101, the area calculation unit 102, the evaluation target identification unit 103, and the activity evaluation unit 104 are not limited to being realized by software, but may also be realized by hardware such as an ASIC (Application Specific Integrated Circuit).

[0019] The skeleton detection unit 101 is an example of a detection unit. The skeleton detection unit 101 detects the skeleton of each person from an image of multiple people captured by the camera device 2. The skeleton detection unit 101 receives images from the camera device 2 via the LAN 90. The skeleton detection unit 101 detects the coordinates of the worker's skeleton by image analysis using, for example, general-purpose AI (Artificial Intelligence).

[0020] For example, the skeleton detection unit 101 stores the coordinates of multiple points on each person's skeleton as time-series skeleton data 130 in the HDD 13. The skeleton data 130 indicates, for example, the coordinates of the nose, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left hip, and right hip of each person #1 to #N (N: positive integer) in the image for each frame number of the image. When the horizontal and vertical directions of the rectangular frame of the image are defined as the X axis and the Y axis, respectively, the coordinates are indicated by the X coordinate and the Y coordinate (x, y). There is no limitation on the position of the origin.

[0021] The area calculation unit 102 is an example of a calculation unit. The area calculation unit 102 calculates the area of ​​a substantially rectangular region having an outline passing through four points on the skeleton of each person: the left and right shoulders and the left and right hips. This region corresponds to the torso of each person. The shape of the torso is, for example, a rectangular shape with corners at the four points described above, but is not limited thereto. For example, it may be a shape with four rounded corners passing through each of the four points. The camera device 2 is positioned substantially in front of the worker to be evaluated so that the work movements of the worker to be evaluated are captured in the image. Therefore, of the multiple people in the image, at least the torso of the worker to be evaluated can be clearly captured. The area calculation unit 102 acquires the coordinates of the four points from the skeleton data 130 and calculates the area using a geometric calculation method. The torso is an example of a substantially rectangular region.

[0022] The evaluation target identification unit 103 is an example of an identification unit. The evaluation target identification unit 103 identifies the worker to be evaluated from multiple people based on the area of ​​the torso. The area of ​​the torso is larger for people closer to the installation position of the camera device 2 and smaller for people farther from the installation position of the camera device 2. In other words, the area is determined according to the distance between the camera device 2 and the person. Therefore, the evaluation target identification unit 103 can identify the worker to be evaluated when the area of ​​the torso is within a range according to the distance between the camera device 2 and the worker to be evaluated.

[0023] The evaluation target identification unit 103 identifies a person whose torso area is larger than a threshold value S1 as a worker to be evaluated. Therefore, by appropriately setting the threshold value S1 according to the distance between the camera device 2 and the worker to be evaluated, a person who appears behind the worker to be evaluated in the image from the camera device 2 is excluded from the evaluation target. Note that the threshold value S1 is an example of a first threshold value.

[0024] Furthermore, the evaluation target identification unit 103 identifies a person whose torso area is larger than a threshold value S2 (>S1) as a worker to be evaluated. Therefore, by appropriately setting the threshold value S2 according to the distance between the camera device 2 and the worker to be evaluated, a person who appears in front of the worker to be evaluated in the image from the camera device 2 is excluded from the evaluation target. Note that the threshold value S2 is an example of a second threshold value.

[0025] Therefore, the evaluation target identification unit 103 can easily identify the worker to be evaluated from the area of ​​the torso, which allows the task evaluation unit 104 to appropriately evaluate the task.

[0026] The task evaluation unit 104 is an example of an evaluation unit. The task evaluation unit 104 evaluates the task of a worker based on changes over time in the coordinates of the skeleton of the worker being evaluated in the image. The task evaluation unit 104 acquires each coordinate of the skeleton of the worker being evaluated from the skeleton data 130, and determines the worker's movements from the changes over time in each coordinate. For example, the task evaluation unit 104 determines whether the worker's movements match predetermined movements, and records the determination result as evaluation data 131.

[0027] (Example of identifying the worker to be evaluated) 2 is a diagram showing an example of an image Ga of the camera device 2 for explaining a method for identifying a worker to be evaluated. The X-axis and Y-axis in the image Ga correspond to the horizontal and vertical directions of the paper, respectively. This also applies to the subsequent drawings.

[0028] Image Ga shows, for example, a workplace before work begins. The camera device 2 is installed in a position where, for example, the worker Ha to be evaluated is positioned approximately in the center of image Ga, and is a sufficient distance away from the workbench 80 of the worker Ha so as not to interfere with the work of the worker Ha. Therefore, the image Ga also shows other workers Hb and people Hc and Hd in addition to the worker Ha. Examples of people Hc and Hd include, but are not limited to, visitors and supervisors of the workplace. The worker Hb and person Hc who are not the subject of evaluation are located behind the worker Ha, and the person Hd who is not the subject of evaluation is located in front of the worker Ha.

[0029] The skeleton detection unit 101 detects the coordinates of the skeletons of the workers Ha, Hb and the persons Hc, Hd from the image Ga. As the coordinates of the skeleton of the worker Ha, for example, the coordinates of the position P0 of the nose of the worker Ha, the position P23 of the left shoulder, the position P13 of the right shoulder, the position P22 of the left elbow, the position P12 of the right elbow, the position P21 of the left hand, the position P11 of the right hand, the position P32 of the left waist, and the position P31 of the right waist can be mentioned. Also, although the symbols are omitted, the coordinates of the skeletons are similarly detected for the other workers Hb and the persons Hc, Hd. The positions P0, P11 to P13, P21 to P23, P31, P32 are shown only in the image Ga, and the illustration is omitted in the images shown later.

[0030] The skeleton detection unit 101 records each of the coordinates of the skeletons of the workers Ha, Hb and the persons Hc, Hd in the skeleton data 130. The area calculation unit 102 calculates the areas Sa to Sd of the torso parts of the workers Ha, Hb and the persons Hc, Hd from the skeleton data 130. For example, the area calculation unit 102 calculates the area Sa from the four coordinates of the position P23 of the left shoulder, the position P13 of the right shoulder, the position P32 of the left waist, and the position P31 of the right waist for the skeleton of the worker Ha. Also, the areas Sb to Sd of the torso parts of the other workers Hb and the persons Hc, Hd are calculated in the same way. The area calculation unit 102 compares the areas Sa to Sd with the predetermined threshold values S1, S2.

[0031] The threshold values S1, S2 define a range including the area Sa of the torso part of the worker Ha to be evaluated, based on, for example, the distance between the camera device 2 and the worker Ha to be evaluated. For this reason, for the area Sa, S1 < Sa < S2 holds, but the areas Sb, Sc are below the threshold value S1, and the area Sd is above the threshold value S2. Therefore, the evaluation target specifying unit 103 specifies the worker Ha to be evaluated by determining that S1 < Sa < S2 holds.

[0032] Furthermore, the area calculation unit 102 may calculate the time average of the areas Sa to Sd according to the time change of the coordinates in the skeletal image Ga. In this case, the evaluation target identification unit 103 determines whether the time average of the areas Sa to Sd is within the range of the thresholds S1 and S2, as described above. This reduces the influence of time changes in the positions and orientations of the workers Ha, Hb and the people Hc, Hd on the calculation of the areas Sa to Sd.

[0033] (Example of work actions) Fig. 3 is a diagram showing an example of images G1b and G2b captured by the camera device 2 of the working motion of the worker Ha. In Fig. 3, the same components as those in Fig. 2 are given the same reference numerals, and the description thereof will be omitted.

[0034] In this example, the work operation using the part C includes, but is not limited to, an operation of assembling the part C acquired in the acquisition operation of Fig. 2 to the work W. Examples of the work operation include an operation of painting the part C, an operation of welding the part C to the work W, and an operation of inspecting the part C.

[0035] The work evaluation unit 104 identifies the work motion of the worker Ha based on the change over time in the coordinates of each part of the worker Ha's skeleton identified by the evaluation target identification unit 103. Image G1b shows the worker Ha holding the workpiece W with his left hand and aligning the assembly position of the part C with his right hand, and image G2b shows the worker Ha assembling the part C to the workpiece W. The work evaluation unit 104 determines the work motion by pattern matching of the change over time in the coordinates of the skeleton in images G1b and G2b and the coordinates of the skeleton in images of other frames not shown. The work evaluation unit 104 evaluates whether the work motion is a specified motion by pattern matching and generates evaluation data 131.

[0036] The task evaluation unit 104 distinguishes various types of task movements in addition to those described above. This distinction can be made using, for example, AI. For example, the task evaluation unit 104 distinguishes and evaluates the type of task using a machine-learned discrimination model based on supervised learning, which receives the coordinates of the skeleton in images G1b and G2b as input and outputs the type of task movement.

[0037] Specifically, in the learning process of the discriminant model, the type of worker's movement is annotated based on the time change in the skeletal coordinates of the training data. The discriminant model is a neural network that mathematically models human brain function, and determines the weighting coefficients of the activation function of the part corresponding to the neuron based on machine learning. In this way, the task evaluation unit 104 can discriminate work movements with high accuracy by using a discriminant model that has been machine-learned using supervised learning.

[0038] (Image masking example) Fig. 4 is a diagram showing an example of an image Gc that has been subjected to mask processing. In Fig. 4, the same components as those in Fig. 2 are given the same reference numerals, and their description will be omitted.

[0039] The task evaluation unit 104 defines a task area R in the image Gc where the task is performed based on changes over time in the coordinates of the skeleton of the worker Ha. For example, when the task evaluation unit 104 identifies a specific task movement, it defines the task area R based on the range in which the coordinates of each part of the skeleton have changed during that task movement. The task evaluation unit 104 masks areas in the image Gc other than the task area R, as indicated by the hatching.

[0040] The skeleton detection unit 101 excludes areas in the image Gc other than the working area R from the skeleton detection range. In other words, the skeleton detection unit 101 does not detect skeletons from areas in the image Gc other than the working area R. This reduces the load on the skeleton detection unit 101 to detect the skeletons of the worker Hb and persons Hc and Hd who are not the target of evaluation.

[0041] (Example of work evaluation server operation) 5 is a flowchart showing an example of the operation of the task evaluation server 1. First, the skeleton detection unit 101 receives image data from the camera device 2 via the LAN 90 (step St1). Next, the skeleton detection unit 101 detects the skeletons of each of the multiple people #1 to #N from the image of the image data (step St2). Next, the skeleton detection unit 101 records the skeleton data 130 for each of the people #1 to #N (step St3).

[0042] Next, the area calculation unit 102 selects a person #i (i: an integer from 1 to N) (step St4). Next, the area calculation unit 102 calculates the area S from the four coordinates of the position P23 of the left shoulder, the position P13 of the right shoulder, the position P32 of the left waist, and the position P31 of the right waist of the person #i being selected from the skeletal data 130 (step St5).

[0043] Next, the evaluation target identification unit 103 compares the area S with the threshold values S1 and S2 respectively (step St6). When S1 < S < S2 holds (Yes in step St6), the evaluation target identification unit 103 sets the person #i being selected as the operator to be evaluated (step St7). Next, the work evaluation unit 104 evaluates the operator to be evaluated (step St8). Note that an example of the evaluation operation will be described later. After that, this operation ends.

[0044] Also, when S1 < S < S2 does not hold (No in step St6), the area calculation unit 102 determines the presence or absence of unselected persons #1 to #N (step St9). When there are unselected persons #1 to #N (Yes in step St9), the operations after step St4 are performed again. At this time, the area calculation unit 102 selects another person #i. Also, when there are no unselected persons #1 to #N (No in step St9), this operation ends.

[0045] In step St6 above, the area S is compared with the threshold values S1 and S2. However, when there are no other persons in front of the operator to be evaluated in the image of the camera device 2, the area S may be compared with only the threshold value S1. In this case, when S1 < S holds, the operator to be evaluated is identified.

[0046] FIG. 6 is a flowchart showing an example of the work evaluation operation. This operation is executed in step St8 of FIG. 7.

[0047] First, the skeleton detection unit 101 receives image data from the camera device 2 via the LAN 90 (step St11). Next, the skeleton detection unit 101 detects the skeletons of one or more persons #1 to #N from the image of the image data (step St12). Next, the skeleton detection unit 101 records the skeleton data 130 of the persons #1 to #N (step St13).

[0048] Next, the task evaluation unit 104 compares the displacement of the skeletal coordinates of the worker to be evaluated within a predetermined period with a threshold value TH (step St14). For example, the task evaluation unit 104 calculates the displacement of the skeletal coordinates for a predetermined number of frames. If the displacement of the skeletal coordinates of the worker to be evaluated within the predetermined period is greater than the threshold value TH (Yes in step St14), the evaluation target identification unit 103 determines that the worker to be evaluated has left the workplace and performs the identification process again (step St20). At this time, the evaluation target identification unit 103 re-identifies the worker using the method shown in FIG. 5. If the re-identification is complete (Yes in step St21), the operation of step St19, which will be described later, is performed. If the re-identification is not complete (No in step St21), the operation of step St20 is performed again.

[0049] If the displacement of the skeletal coordinates of the worker to be evaluated within the predetermined period is equal to or less than the threshold value TH (No in step St14), the evaluation target identification unit 103 identifies the worker's work motion based on the time change of the skeletal coordinates in the image (step St15). Next, the work evaluation unit 104 evaluates the work by comparing the identified work motion with a specified motion (step St16). At this time, the work evaluation unit 104 records the evaluation result in the evaluation data 131.

[0050] Next, if the work area R in the image has not been defined (No in step St17), the work evaluation unit 104 defines the work area R based on the time change in the coordinates of the worker's front body, and performs masking of the area other than the work area R (step St18). The skeleton in the masked area will no longer be detected in step St12. Also, if the work area R in the image has been defined (Yes in step St17), step St18 is not performed.

[0051] Next, the task evaluation unit 104 determines whether all tasks have been completed based on the skeleton data 130 (step St19). If there are any unfinished tasks (No in step St19), the processes from step St11 onward are performed again, and if all tasks have been completed (Yes in step St19), this process ends. This is how the task evaluation server 1 operates.

[0052] The above-described embodiment is a preferred example of the present invention, but the present invention is not limited to this and can be modified in various ways without departing from the spirit of the present invention. [Explanation of symbols]

[0053] 1 Work evaluation server (work evaluation device), 10 CPU, 101 skeleton detection unit (detection unit), 102 area calculation unit (calculation unit), 103 evaluation target identification unit (identification unit), 104 work evaluation unit (evaluation unit), 130 skeleton data, Ha, Hb worker, Hc, Hd person

Claims

1. a detection unit that detects skeletons of each of a plurality of people from an image of the people; a calculation unit that calculates a time average of an area of ​​a substantially rectangular region having an outline passing through four points, i.e., the left and right shoulders and the left and right hips, of the skeleton of each person in accordance with changes in coordinates of the four points in the image over time; an identification unit that identifies, from among the plurality of persons, a person whose time average of the area is greater than a first threshold value as a worker to be evaluated; an evaluation unit that evaluates the work of the worker based on time changes in the coordinates of the skeleton of the worker in the image, Work evaluation device.

2. the identification unit identifies, from among the plurality of persons, a person whose time average of the area is smaller than a second threshold, as a worker to be evaluated. The task evaluation device according to claim 1 .

3. the evaluation unit defines a work area in the image where the work is performed based on a time change in coordinates of the skeleton of the worker; the detection unit excludes an area other than the working area in the image from the skeleton detection range; The task evaluation device according to claim 1 or 2.

4. the evaluation unit uses a machine-learned discrimination model through supervised learning that receives input of coordinates of the skeleton in the image and outputs the type of the task, and discriminates and evaluates the type of task based on changes in the coordinates of the skeleton over time. The task evaluation device according to claim 1 or 2.

Images