Workload assessment device and annotation support device

Abstract

To provide: a workload determination device capable of accurately determining a worker's load even in a transient state of work; and an annotation support device.SOLUTION: A captured image acquisition part 14 acquires a captured moving image obtained by capturing an image of a work area in which a worker works, by a camera 12, and a determination part 16 determines a worker's load in the captured image by using a worker load determination model 18 having already learned by machine learning. For example, frame images are extracted from the captured image, and each frame image is input to the worker load determination model 18, thereby determining the worker's workload in the frame image. The worker load determination model 18 determines the worker's load based on a space in which a part of the worker preset according to the work content is located.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a workload determination device and an annotation support device. [Background technology]

[0002] Patent Document 1 proposes a workload analysis device that visualizes the workload of a worker in a work process. In detail, the workload analysis device includes a sensor for monitoring the state of the worker, a determination unit that determines whether the contractor is working or not based on data acquired by the sensor, and a display unit that displays the contractor's workload based on the time the worker is working and the time not working during a predetermined period.

[0003] Patent Document 1 also discloses determining whether a worker is working based on the worker's position, and determining whether a worker is working based on the total amount of movement of a specific part of the worker. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-125183 Summary of the Invention [Problem to be solved by the invention]

[0005] When making binary judgments, such as whether or not a worker is working, based on the worker's position or the amount of movement of a specific part of the worker's body, there is a possibility of incorrect judgment in transitional states, such as when a worker goes from not working to starting work, or when a worker goes from working to finishing work.

[0006] For example, a model that determines worker workload from captured images can be used to determine worker workload. In this case, when creating training data for machine learning of the model, an annotation process is performed to label workers in the captured images, but there is a risk of assigning incorrect labels. In the transient state of human behavior, even people with knowledge and expertise of the work being assessed find it difficult to define classifications when assigning labels. If an incorrect label is assigned during annotation and machine learning is performed, the workload will be incorrectly determined.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide a workload determination device and an annotation support device that can determine the workload of a worker with high accuracy even during a transient state of work. [Means for solving the problem]

[0008] The workload assessment device according to the first aspect includes a photographed image acquisition unit that acquires a photographed image of a worker taken by a camera, and a predetermined image acquisition unit that acquires a predetermined image according to the work content. 、 Worker's body part and the relevant part Based on the space where the worker is located, work and a determining unit that determines the load.

[0009] According to the first aspect, even when the work is in a transient state, it is possible to determine whether the load is high or low by identifying the space in which the worker's body parts, which are set according to the work content, are located, so that the worker's load can be determined with high accuracy even when the work is in a transient state.

[0010] A workload determination device according to a second aspect is the workload determination device according to the first aspect, wherein the part is determined based on the work content of the worker, Other parts of the worker This is a part that moves more than the other parts.

[0011] According to the second aspect, it is possible to set a part that reflects the load of the worker, and it is possible to determine the load of the worker more accurately than by setting a part with a small amount of movement.

[0012] A workload determination device according to a third aspect is the workload determination device according to the first or second aspect, wherein the determination unit is In the case of a transient state in which a part is moving from a preset high load area to a low load area, or from a low load area to a high load area, The load is determined as the load set for the area.

[0013] According to the third aspect, it is possible to determine the load of the worker with high accuracy even when the worker is in a transient state.

[0014] The workload assessment device according to a fourth aspect is the workload assessment device according to any one of the first to third aspects, wherein the assessment unit assesses the workload of the worker in the captured image using a worker workload assessment model that has been trained by machine learning, and outputs the workload of the worker assessed based on a space in which a predetermined part of the worker's body is located according to the work content.

[0015] According to the fourth aspect, it is possible to obtain the load on the worker, which is determined based on the space in which the body part of the worker, which is set in advance depending on the type of work, is located, from the captured image.

[0016] An annotation support device according to a fifth aspect includes: a captured image acquisition unit that acquires a captured image of a worker by a camera; an extraction unit that extracts a worker image in which a worker is present from the captured image acquired by the captured image acquisition unit; and an extraction unit that extracts the worker from the worker image extracted by the extraction unit; Pre-set according to the work content The system includes an identification unit that identifies the worker's body part and determines whether the identified body part is located in a pre-set high-stress area or a pre-set low-stress area, and a label assignment unit that assigns a label indicating the workload to the worker in the worker image according to the identification result of the identification unit.

[0017] According to the fifth aspect, it is possible to generate learning data for determining the load on a worker with high accuracy even in a transient state of work. [Effects of the Invention]

[0018] As described above, according to the present invention, it is possible to provide a workload determination device and an annotation support device that can determine the workload of a worker with high accuracy even in a transient state of work. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a schematic configuration of a workload determination device according to an embodiment of the present invention; [Figure 2] FIG. 1 is a block diagram showing a schematic configuration of a computer. [Figure 3] FIG. 10 is a diagram showing an example of a frame image of a moving image captured on an assembly line. [Figure 4] FIG. 10 is a diagram for explaining an example of determining the load of a worker in a transient state. [Figure 5] FIG. 1 is a block diagram showing a schematic configuration of an annotation support device. [Figure 6] 10 is a flowchart showing an example of the flow of processing performed by the annotation support device according to the present embodiment. [Figure 7] 4 is a flowchart showing an example of a flow of processing performed by the workload determination device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a block diagram showing a schematic configuration of a workload determination device according to this embodiment.

[0021] As shown in Fig. 1, the workload assessment device 10 according to this embodiment includes a photographed image acquisition unit 14, a determination unit 16, a worker workload assessment model 18, and an output unit 20. In this embodiment, the photographed image acquisition unit 14, the determination unit 16, and the output unit 20 are described as functions executed by a computer 11 (see Fig. 2). Fig. 2 is a block diagram showing a schematic configuration of the computer.

[0022] 2, the computer 11 has a general computer configuration including a CPU (Central Processing Unit) 11A, a ROM (Read Only Memory) 11B, a RAM (Random Access Memory) 11C, a storage 11D, an interface (I / F) 11E, and a bus 11F. The CPU 11A loads programs such as a workload determination program stored in the ROM 11B into the RAM 11C and executes them, thereby functioning as a photographed image acquisition unit 14, a determination unit 16, and an output unit 20.

[0023] The captured image acquisition unit 14 acquires captured images of a moving image of the work area where the worker works, captured by the camera 12. The camera 12 is fixedly placed in the process for which the workload is to be determined, and captures images of the worker.

[0024] The determination unit 16 determines the workload of the worker in the captured images acquired by the captured image acquisition unit 14 using a worker workload determination model 18 that has been trained by machine learning. For example, frame images are extracted from the captured images and each frame image is input to the worker workload determination model 18, thereby determining the workload of the worker in the frame images. In this embodiment, the worker workload determination model 18 determines the workload of the worker based on the space in which the worker's body parts, which are preset according to the work content, are located.

[0025] The output unit 20 performs a process of outputting the judgment result of the worker's workload judged by the judgment unit 16. As an output method, for example, a process of displaying the judgment result of the judgment unit 16 on a display unit or the like is performed.

[0026] Here, an annotation procedure for creating learning data for creating the worker workload judgment model 18 will be described. As an example, a case will be described in which learning data is created using video captured on an assembly line for vehicles, etc. Fig. 3 shows an example of frame images of the video captured on the assembly line.

[0027] A camera is installed on the assembly line for which the load on workers is to be determined, and video images are obtained.

[0028] Still images are cut out from the captured moving images frame by frame, and frames in which the worker appears are extracted from the cut out still images.

[0029] For the frames in which the extracted workers appear, annotation work is performed to classify the workers' states into "high load" and "low load."

[0030] Since it is particularly difficult to classify human movements into binary values ​​during a transient state, in this embodiment, the system determines which area of ​​the image a body part of the worker, which has been preset according to the type of work, is located in. In other words, the system determines whether the body part of the worker, which has been preset according to the type of work, is located in a high-stress area or a low-stress area from the image, and assigns a label to it.

[0031] For example, in the case of the assembly line of FIG. 3, it is determined whether the position of a body part such as a worker's hand is in a predetermined high-stress area or a low-stress area. In the example of FIG. 3, the area on the assembly line 22 is the high-stress area, and the area outside the assembly line 22 (the diagonally hatched area in FIG. 3) is the low-stress area. In this case, workers on the assembly line 22 are labeled as high-stress, and workers outside the assembly line 22 are labeled as low-stress. Also, as shown in FIG. 3, in the transient state where a worker outside the assembly line 22 moves onto the assembly line 22, the class classification is determined, for example, by the position of the hand. In the example of FIG. 3, the hand of the worker on the left in the lower image is located in the high-stress area, so it is labeled as high-stress.

[0032] The body parts of the worker that are used to determine the classification are set to the parts of the worker that move the most depending on the work content. As an example, the body part of the worker that moves the most is set. Specifically, if the work involves holding an object, the position of the hands is used to determine the classification, and if the whole body is moving, the head is often moved, so the position of the head is used to determine the classification. Note that data on which body parts to use for the classification may be stored in advance, or the body part to use for the classification may be decided on the spot. Weighting and priority of each body part may also be set, allowing multiple selections.

[0033] The worker's body parts, which are set according to the work content, are identified as being in a transient state over several frames before and after, and a classification is determined according to the transition direction of the transient state. For example, if the transient state is from high load area A to low load area B, it is determined to be in low load area B because it is in the middle of transitioning to low load area B, and if the transient state is from low load area B to high load area A, it is determined to be in high load area A because it is in the middle of transitioning to high load area A.

[0034] To ensure accuracy, the classification may be determined based on the amount of movement of the worker's body parts, which are set according to the type of work.For example, in the example of Figure 4, in the transient state on the dashed line where the worker moves from low-stress area B to high-stress area A, the movement is from low-stress area B to high-stress area A, and the amount of movement is also increasing, so it is determined to be a high-stress point.

[0035] In the above annotation procedure, the annotation work has been described as a work performed by a person, but the annotation work may be performed using an annotation support device shown in Fig. 5 to create learning data. Fig. 5 is a block diagram showing a schematic configuration of the annotation support device 30. The annotation support device 30 in Fig. 5 will now be described in detail. Note that the same functional parts as those in the workload determination device 10 will be described using the same reference numerals.

[0036] 5, the annotation support device 30 includes a captured image acquisition unit 14, an extraction unit 32, a setting unit 34, an identification unit 36, and a label assignment unit 38. Note that, similar to the workload assessment device 10, the captured image acquisition unit 14, the extraction unit 32, the setting unit 34, the identification unit 36, and the label assignment unit 38 will be described as functions executed by the computer 11. That is, the CPU 11A loads a program, such as a workload assessment program stored in the ROM 11B, into the RAM 11C and executes it, thereby functioning as the captured image acquisition unit 14, the extraction unit 32, the setting unit 34, the identification unit 36, and the label assignment unit 38.

[0037] Although the workload determination device 10 and the annotation support device 30 are shown as the same computer 11 in FIG. 2, they may be separate computers.

[0038] Similar to the workload assessment device 10, the captured image acquisition unit 14 acquires moving images of the work area where the worker works, captured by the camera 12. The camera 12 is fixedly placed in the process for which the workload is to be assessed, and captures images of the worker.

[0039] The extraction unit 32 extracts frame images as worker images in which a worker is present from the captured images acquired by the captured image acquisition unit 14. For example, by using deep learning technology for object detection such as Yolo (You only look once) or SSD (Single Shot MultiBox Detector), which are well-known image recognition technologies, the extraction unit 32 recognizes the worker from the captured images and extracts frames in which the worker is present using a model trained in advance by machine learning or the like.

[0040] The setting unit 34 sets high-stress areas and low-stress areas, and sets the worker's body parts according to the work content. For example, the setting of high-stress areas and low-stress areas is performed by accepting a selection result of each of the high-stress areas and low-stress areas in the captured image, thereby setting the areas in the captured image as high-stress areas and low-stress areas. Furthermore, the setting of body parts according to the work content is performed by, for example, accepting a selection result of which body part of the worker to focus on according to the work content of the process whose workload is to be assessed, thereby setting the worker's body parts according to the work content that are to be focused on for assessing the workload.

[0041] The identification unit 36 ​​extracts the recognized worker from the frame image extracted by the extraction unit 32, and performs processing to identify the worker's body part according to the work content set by the setting unit 34. Then, it identifies whether the identified worker's body part is located in the high-stress area or the low-stress area set by the setting unit 34.

[0042] The labeling unit 38 assigns a label indicating the workload to the worker in the frame image according to the identification result by the identification unit 36. In this embodiment, a label indicating high workload or low workload is assigned. By creating multiple labeled frame images, learning data can be generated.

[0043] The learning data to which the labels are assigned by the label assignment unit 38 is used as training data, and the worker workload determination model 18 is constructed by training the learning device 40 by machine learning such as a neural network.

[0044] This makes it possible to determine the workload of the worker in the captured image by inputting the captured image into the worker load determination model 18. That is, by inputting the captured image of the moving image captured by the camera 12 into the workload determination device 10, the determination unit 16 can use the worker load determination model 18 to determine the workload of the worker in the captured image.

[0045] Next, a specific process performed by the annotation support device 30 according to this embodiment configured as described above will be described. Fig. 6 is a flowchart showing an example of the flow of the process performed by the annotation support device 30 according to this embodiment. Note that the process in Fig. 6 starts when, for example, a user performing annotation work operates the computer 11 to instruct the start of annotation work.

[0046] In step 100, the CPU 11A acquires a captured image, and the process proceeds to step 102. That is, the captured image acquisition unit 14 acquires a captured image of a moving image captured by the camera 12 of the work area where the worker is working.

[0047] In step 102, the CPU 11A extracts a frame image and proceeds to step 104. That is, the extraction unit 32 uses a model that has been trained in advance by machine learning or the like, for example, using a well-known deep learning technique for object detection, to recognize the worker from the captured image and extract a frame image in which the worker is present.

[0048] In step 104, the CPU 11A identifies the worker and the body part, and proceeds to step 106. That is, the identification unit 36 ​​extracts the recognized worker in the frame image extracted by the extraction unit 32, and identifies the body part of the worker according to the work content set by the setting unit 34.

[0049] In step 106, the CPU 11A determines the area of ​​the identified body part, and proceeds to step 108. That is, the identification unit 36 ​​identifies whether the identified body part of the worker is located in the high load area or the low load area set by the setting unit 34.

[0050] In step 108, the CPU 11A assigns a label and the process proceeds to step 110. That is, the label assignment unit 38 assigns a label to the worker in the frame image in accordance with the identification result of the identification unit 36.

[0051] In step 110, the CPU 11A determines whether or not there is a next frame image. If the determination is affirmative, the process returns to step 102 and the above-described processing is repeated. When the determination is affirmative, the series of processing performed by the annotation support device 30 is terminated.

[0052] Next, a specific process performed by the workload determination device 10 for determining workload using the worker workload determination model 18 constructed using the learning data annotated as described above will be described. Fig. 7 is a flowchart showing an example of the flow of the process performed by the workload determination device 10 according to this embodiment. The process in Fig. 7 starts, for example, when an instruction to start workload determination is given by operating the computer 11.

[0053] In step 200, the CPU 11A acquires a captured image, and the process proceeds to step 202. That is, the captured image acquisition unit 14 acquires a captured image of a moving image obtained by capturing with the camera 12 the work area where the worker is working.

[0054] In step 202, the CPU 11A extracts a frame image and proceeds to step 204. That is, the determination unit 16 extracts a frame image from the captured image.

[0055] In step 204, the CPU 11A determines whether or not there is a worker in the extracted frame image. If the worker load determination model 18 does not detect the worker to be determined, the determination is denied and the process proceeds to step 202, where the determination is made on the next frame image. On the other hand, if the determination is affirmative, the process proceeds to step 206.

[0056] In step 206, the CPU 11A determines the workload and proceeds to step 208. That is, the determination unit 16 determines the workload of the worker in the frame image by inputting the frame image into the worker workload determination model 18. In this embodiment, the workload of the worker is determined based on the space in which the worker's body parts, which are set in advance according to the work content, are located, so that the workload of the worker can be determined with high accuracy even in a transient state of the work.

[0057] In step 208, the CPU 11A outputs the determination result, and the process proceeds to step 210. The output unit 20 displays the determination result of the determination unit 16 on, for example, a display unit or the like.

[0058] In step 210, the CPU 11A determines whether or not there is a next frame image. If the determination is affirmative, the process returns to step 202 and the above-described processing is repeated, and when the determination is affirmative, the series of processing performed by the workload determination device 10 is terminated.

[0059] In this way, the workload determination device 10 according to this embodiment determines the workload of a worker based on the space in which the worker's body parts, which are set in advance according to the work content, are located. As a result, even in a transitional state of work, it is possible to determine whether the workload is high or low by identifying the space in which the worker's body parts, which are set in accordance with the work content, are located, so that the workload of a worker can be determined with high accuracy even in a transitional state of work.

[0060] Furthermore, by setting a body part of the worker to be focused on depending on the work content, it is possible to determine the load on the worker in various processes, without being limited to a specific process.

[0061] In the above embodiment, the computer 11 functioning as the workload determination device and the annotation support device 30 may be a personal computer or a computer such as a server. When a server is used, the processes performed by the workload determination device 10 and the annotation support device 30 may be provided as a cloud service.

[0062] In the above embodiment, an example has been described in which the worker load is determined using the machine-learned worker load determination model 18, but the present invention is not limited to this. For example, instead of using a machine-learned model, image processing may be used to identify a space in which a predetermined part of the worker is located according to the work content, and the predetermined worker load in the identified space may be identified.

[0063] Furthermore, although the processing performed by the computer 11 in each of the above embodiments has been described as software processing performed by executing a program, this is not limited to this. For example, the processing may be performed by hardware such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array). Alternatively, the processing may be a combination of both software and hardware. Furthermore, if the processing is software, the program may be stored in various storage media and distributed.

[0064] Furthermore, the present invention is not limited to the above, and it goes without saying that various modifications can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0065] 10 Workload determination device 11 Computer 12 Camera 14. Image acquisition unit 16 Judgment section 18 Worker workload judgment model 30 Annotation Support Device 32 Extraction part 34 Setting section 36 Specific part 38 Labeling section

Claims

1. a captured image acquisition unit that acquires a captured image of the worker taken by a camera; a determination unit that determines the workload of the worker in the captured image acquired by the captured image acquisition unit based on a body part of the worker and a space in which the body part is located, which are set in advance according to the work content; A workload determination device comprising:

2. The workload determination device according to claim 1 , wherein the body part is a body part that moves more than other body parts of the worker during the work of the worker.

3. The workload determination device according to claim 1, wherein the determination unit determines that the load of the part is set to the area to which the part is moved when the part is in a transient state of moving from a predetermined high-load area to a low-load area, or from a low-load area to a high-load area.

4. 2. The workload determination device according to claim 1, wherein the determination unit determines the workload of the worker in the captured image using a worker workload determination model that has been trained by machine learning, and that outputs a determined workload of the worker based on a space in which a predetermined part of the worker's body is located according to the work content.

5. a captured image acquisition unit that acquires a captured image of the worker taken by a camera; an extraction unit that extracts a worker image in which a worker is present from the captured image acquired by the captured image acquisition unit; an identification unit that extracts the worker from the worker image extracted by the extraction unit, identifies a predetermined body part of the worker depending on the work content, and identifies whether the identified body part is located in a predetermined high-stress area or a predetermined low-stress area; a labeling unit that assigns a label indicating a workload to the worker in the worker image according to the identification result of the identification unit; An annotation support device equipped with the above.

Images