Learning model generation method, work analysis apparatus, and work analysis program

The learning model generation method and device capture images from the worker's forehead to estimate posture and motion accurately, addressing the limitations of previous systems by using key point associations and machine learning to analyze work data effectively.

JP2024165569APending Publication Date: 2024-11-28KUBOTA CORP
View PDF 18 Cites 0 Cited by

Patent Information

Application Number
JP2023081855
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing work analysis systems struggle to accurately estimate a person's posture and motion regardless of their location or body orientation, particularly when using wearable cameras attached to hats or canopies, which can fail to capture the necessary images due to head direction or posture.

Method used

A learning model generation method and device that captures images from the forehead of a worker using a wearable camera, associates key points on the median plane of the body with labels, and performs machine learning to estimate posture and motion, regardless of the worker's location or body orientation.

Benefits of technology

The system reliably captures and analyzes work data to estimate posture and motion accurately, irrespective of the worker's position or body orientation, enhancing the accuracy and consistency of work analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165569000001_ABST
    Figure 2024165569000001_ABST
Patent Text Reader

Abstract

To provide a learning model generation method for generating a learning model configured to estimate an attitude of a person who performs a task, regardless of the location where the person performs the work and the orientation and attitude of the body, such as the head of the person, a work analysis apparatus, and a work analysis program.SOLUTION: A learning model generation method includes the steps of: capturing an image, by an imaging unit mounted on the head of a person, from the head toward the front of the person to acquire work data related to a task; associating the acquired work data with labels representing key points of hands of the person, based on a key point of a median plane of the person, to generate teaching data; and generating a learning model which is trained by machine learning using the teaching data, and configured to receive, as input, the acquired work data and output data related to motion estimation of the person.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a learning model generation method, an activity analysis device, and an activity analysis program. [Background technology]

[0002] Patent Document 1 discloses a posture analysis device having a skeleton extraction unit that acquires skeleton data including feature point data indicating the joint positions of a person appearing in the image data by image recognition using the image data as input, a posture model storage unit in which a posture label is associated with each piece of skeleton data, and a posture estimation unit that determines the posture of the person appearing in the image data from the posture labels predetermined in the posture model, based on the skeleton data acquired by the skeleton extraction unit.

[0003] In the posture analysis device described in Patent Document 1, image data is acquired by a video camera capturing an image of a worker as a subject, and is input from the video camera. However, Patent Document 1 does not describe the installation position and shooting direction of the video camera. Therefore, depending on the location where a person works and the direction or posture of the person's body, the video camera may not be able to capture the worker as a subject.

[0004] Patent document 2 discloses a work suitability judgment device including a wearable camera capable of capturing an image of an area approximately equal to or larger than the field of vision of a worker, an object detection means for detecting an object within the image capture area captured by the wearable camera, a hand skeleton detection means for detecting the skeleton of the worker's hand within the image capture area, a motion detection means for detecting the behavior of the worker's hand from time-series changes in the object detected by the object detection means and the skeleton of the worker's hand detected by the hand skeleton detection means, and a work suitability judgment means for determining whether or not at least the object detected by the object detection means and the behavior of the worker's hand detected by the motion detection means approximately match previously learned object and hand behaviors of the worker, and for determining that a specified work has been performed appropriately if they approximately match.

[0005] In the work suitability assessment device described in Patent Document 2, the wearable camera is attached to the top surface of a visor provided at the front of the worker's hat, and captures images of the target body (object) and the skeletal structure of the worker's hands within the imaging area. However, because the wearable camera described in Patent Document 2 is attached to the top surface of a visor provided at the front of the worker's hat, there is a risk that the wearable camera will not be able to capture images of at least either the target body (object) or the skeletal structure of the worker's hands, depending on the direction or posture of the worker's head. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 6825041 [Patent Document 2] Patent Publication No. 2022-106155 Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a learning model generation method, a work analysis device, and a work analysis program that generate a learning model that can estimate the posture of a person when performing work, regardless of the place where the person performs the work, or the orientation and posture of the person's body, such as the head. [Means for solving the problem]

[0008] A first aspect of the present invention is a learning model generation method for generating a learning model to be used in a task analysis device that analyzes tasks performed by a person, comprising the steps of: capturing an image from the head toward the front of the person using an imaging unit attached to the person's head to acquire task data related to the task; generating teacher data by matching the acquired task data with labels that indicate key points of the person's hands when a key point on the person's median surface is used as a starting point; and performing machine learning using the teacher data to generate a learning model that uses the acquired task data as input and data related to an estimation of the person's posture as output.

[0009] A second aspect of the present invention is a task analysis device that analyzes tasks performed by a person, comprising: an imaging unit that is worn on the person's head and captures an image from the head toward the front of the person to acquire task data related to the task; a teacher data generation unit that generates teacher data by matching the task data acquired by the imaging unit with labels that indicate key points on the person's hands when a key point on the person's median surface is used as a starting point; and a machine learning unit that performs machine learning using the teacher data to generate a learning model that uses the task data acquired by the imaging unit as an input and data related to an estimation of the person's posture as an output.

[0010] A third aspect of the present invention is a task analysis program executed by a computer of a task analysis device that analyzes tasks performed by a person, the task analysis program causing the computer to execute the following steps: capturing an image from the head toward the front of the person using an imaging unit attached to the person's head to acquire task data related to the task; generating teacher data by matching the acquired task data with labels indicating key points of the person's hands when a key point on the person's median surface is used as a starting point; and performing machine learning using the teacher data to generate a learning model that uses the acquired task data as input and data related to an estimation of the person's posture as output. Effect of the Invention

[0011] According to the present invention, it is possible to provide a learning model generation method, a work analysis device, and a work analysis program for generating a learning model that can estimate the posture of a person when performing work, regardless of the place where the person performs the work or the orientation and posture of the person's body, such as the head. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of a main part of a work analysis device according to a first embodiment of the present invention. [Diagram 2] 4 is a schematic diagram illustrating a first specific example of work data captured and acquired by an imaging section of the present embodiment. FIG. [Diagram 3] 11 is a schematic diagram illustrating a second specific example of work data captured and acquired by the imaging section of the present embodiment. FIG. [Figure 4] 1 is a flowchart illustrating a learning model generation method according to the present embodiment. [Diagram 5] 5A to 5C are schematic diagrams illustrating specific examples of work data captured and acquired by an imaging section of the present embodiment. [Figure 6] FIG. 13 is a schematic diagram illustrating labels that represent relative distances between keypoints as discrete values ​​in a plurality of divisions. [Figure 7] 1 is a flowchart illustrating a learning model generation method according to the present embodiment. [Figure 8] 11 is a table showing an example of labels in which the relative distance of each keypoint is expressed as a discrete value in a plurality of intervals. [Figure 9] 11 is a table showing an example of labels expressing the area of ​​each bounding box as discrete values ​​in multiple segments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiments described below are preferred specific examples of the present invention, and therefore various technically preferable limitations are applied to them, but the scope of the present invention is not limited to these aspects unless otherwise specified in the following description to the effect that the present invention is limited to them. In addition, in each drawing, similar components are given the same reference numerals, and detailed descriptions thereof are omitted as appropriate.

[0014] FIG. 1 is a block diagram showing the configuration of a main part of a work analysis device according to a first embodiment of the present invention. FIG. 2 is a schematic diagram showing a first specific example of work data captured and acquired by the imaging section of this embodiment. FIG. 3 is a schematic diagram illustrating a second specific example of the work data captured and acquired by the imaging section of the present embodiment.

[0015] The work analysis device 2 according to the first embodiment of the present invention generates a learning model for estimating the posture of a person performing work (hereinafter, for convenience of explanation, referred to as "worker") and analyzes the work performed by the worker. In this specification, "work" may be, for example, indoor manufacturing work, outdoor plumbing work, farm work, etc. However, the work analyzed by the work analysis device 2 is not limited to the above-mentioned work.

[0016] 1, the work analysis apparatus 2 according to this embodiment includes a control unit 3, a storage unit 4, a communication unit 5, and an imaging unit 62. As shown in FIG. 1, the work analysis apparatus 2 may further include a worker terminal 61 and a display unit 7.

[0017] The imaging unit 62 is, for example, a wearable camera having an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). An example of the imaging unit 62 is a UVC (USB Video Class) camera. As shown in Fig. 1, the imaging unit 62 is electrically connected to the worker terminal 61 via a cable 64, and transmits captured image data to the computer 21 of the work analysis apparatus 2 via the worker terminal 61.

[0018] The worker terminal 61 is a portable terminal device such as a smartphone or a tablet computer used by a worker. The worker terminal 61 may have a display including a touch panel capable of detecting touch by a worker's finger, for example.

[0019] The imaging unit 62 is attached to the head of the worker and captures an image from the head of the worker toward the front of the worker. Specifically, the imaging unit 62 is attached to the forehead of the worker and captures an image from the forehead toward the front of the worker to obtain work data related to the work in front of the worker. The work data is image data obtained by imaging the worker with the imaging unit 62. In this specification, "image data" or "work data" includes not only data related to a single still image captured by the imaging unit 62 at any timing, but also data related to a still image obtained by extracting a part of a set of continuous images (i.e., a video or video).

[0020] 3, for example, the imaging unit 62 is attached to the brim of a helmet 63 worn by the worker on the head, captures an image from the worker's forehead toward the front of the worker, and obtains work data related to the work in front of the worker. Alternatively, the imaging unit 62 may be attached to the brim of a hat (not shown) worn by the worker on the head, captures an image from the worker's forehead toward the front of the worker, and obtains work data related to the work in front of the worker. Alternatively, the imaging unit 62 may be attached to the brim of a belt (not shown) worn by the worker on the head, captures an image from the worker's forehead toward the front of the worker, and obtains work data related to the work in front of the worker.

[0021] 3, the imaging unit 62 is electrically connected to the worker terminal 61 via a cable 64. The work data acquired by the imaging unit 62 by capturing an image of the area in front of the worker is transmitted to the computer 21 of the work analysis device 2 via the worker terminal 61 and the communication unit 5, and is stored in the memory unit 4.

[0022] 1, the computer 21 of the work analysis apparatus 2 has a control unit 3 and a storage unit 4, and executes various calculations and processes by reading out a program 44 stored in the storage unit 4. The "computer" here is not limited to a personal computer, but also includes an arithmetic processing unit, a microcomputer, etc. included in information processing equipment, and is a general term for equipment and devices that can realize the functions of the present invention by means of a program.

[0023] The program 44 of this embodiment is an example of the "task analysis program" of the present invention. The program 44 includes an image processing program for image processing, an image recognition program for image recognition, a sequence program for generating teacher data and a learning model, etc. Note that the program 44 is not limited to being stored in the storage unit 4, and may be stored in advance in a computer-readable storage medium and distributed, or may be downloaded to the work analysis device 2 via a network.

[0024] The control unit 3 is, for example, a CPU (central processing unit) and reads out a program 44 stored in the storage unit 4 to execute various calculations and processes. The control unit 3 has a teacher data generation unit 31 and a machine learning unit 32. The teacher data generation unit 31 and the machine learning unit 32 are realized by the computer 21 executing the program 44 stored in the storage unit 4. Note that the teacher data generation unit 31 and the machine learning unit 32 may be realized by hardware or a combination of hardware and software.

[0025] The teacher data generation unit 31 generates teacher data 42 by matching the work data 41 as input data acquired by the imaging unit 62 and stored in the memory unit 4 with labels indicating the key points of each part of the worker when the key point on the worker's median surface is used as the starting point, and stores the teacher data 42 in the memory unit 4.

[0026] 2, the teacher data generating unit 31 generates teacher data 42 by associating task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with labels indicating key points of the hands of the worker 60 when a key point on the median plane 65 of the worker 60 is taken as the starting point, and stores the teacher data 42 in the storage unit 4. The key point on the median plane 65 of the worker 60 is specifically a key point K0 on the navel of the worker 60.

[0027] 2, the key points on the hand of the worker 60 include a key point K5 on the left wrist of the worker 60 and a key point K6 on the right wrist of the worker 60. The key points on the hand of the worker 60 may include key points on the fingers of the left hand of the worker 60 and key points on the fingers of the right hand of the worker 60. The key points on the hand of the worker 60 may also include key points on the palm of the left hand of the worker 60 and key points on the palm of the right hand of the worker 60. The key points on the hand of the worker 60 may also include key points on the back of the left hand of the worker 60 and key points on the back of the right hand of the worker 60.

[0028] 2, the teacher data generating unit 31 may generate teacher data 42 by further associating the task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with labels indicating key points of the chest of the worker 60, and store the teacher data 42 in the storage unit 4. In this case, as shown in FIG. 2, the key points of the chest of the worker 60 include a key point K1 on the left chest of the worker 60 and a key point K2 on the right chest of the worker 60.

[0029] 2, the teacher data generating unit 31 may generate teacher data 42 by further associating the task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with labels indicating key points of the arm of the worker 60, and store the teacher data 42 in the storage unit 4. In this case, as shown in FIG. 2, the key points of the arm of the worker 60 include a key point K3 of the left elbow of the worker 60 and a key point K4 of the right elbow of the worker 60.

[0030] Furthermore, the teacher data generating unit 31 may further associate the task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with labels indicating key points of the feet of the worker 60, thereby generating teacher data 42 and storing it in the storage unit 4. In this case, the key points of the feet of the worker 60 include key points of the left foot of the worker 60 and key points of the right foot of the worker 60.

[0031] 2, the teacher data generating unit 31 may further associate the task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with a label indicating a bounding box of the hand of the worker 60, thereby generating teacher data 42 and storing the teacher data 42 in the storage unit 4. In this case, as shown in FIG. 2, the bounding box of the hand of the worker 60 includes a bounding box B1 of the left hand of the worker 60 and a bounding box B2 of the right hand of the worker 60.

[0032] The bounding box of this embodiment is an example of the "area" of the present invention. That is, the characteristic range of the hands and feet of the worker 60 is not limited to a rectangular bounding box, but may be an area showing the shape or outline of the hands and feet. In the following description, a case where the "area" of the present invention is a bounding box is taken as an example.

[0033] Furthermore, the teacher data generating unit 31 may further associate the task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with a label indicating a bounding box of the feet of the worker 60, thereby generating teacher data 42 and storing it in the storage unit 4. In this case, as shown in FIG. 2, the bounding box of the feet of the worker 60 includes a bounding box B3 of the left foot of the worker 60 and a bounding box B4 of the right foot of the worker 60.

[0034] The machine learning unit 32 performs machine learning using the teacher data 42 stored in the storage unit 4, inputs the task data 41 acquired by the imaging unit 62, generates a learning model 43 having data related to posture estimation of the worker 60 as an output, and stores the model in the storage unit 4. That is, the machine learning unit 32 performs machine learning using the teacher data 42 stored in the storage unit 4, generates a learning model 43 for estimating the posture of the worker 60, and stores the model in the storage unit 4. The machine learning unit 32 uses the teacher data 42 stored in the storage unit 4 to generate the learning model 43 by a known algorithm such as a neural network (NN). As a result, when the task data 41 is input as input data, a learning model 43 that outputs data related to posture estimation of the worker 60 is generated.

[0035] The memory unit 4 stores the task data 41 captured by the imaging unit 62 and received via the worker terminal 61 and the communication unit 5, the teacher data 42 generated by the teacher data generation unit 31, the learning model 43 generated by the machine learning unit 32, and a program 44 executed by the computer 21. Examples of the memory unit 4 include a semiconductor memory or a hard disk drive (HDD: Hard Disk Drive) built into the work analysis device 2. Alternatively, examples of the memory unit 4 include various storage media and storage devices connectable to the work analysis device 2, such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a RAM (Random access memory), a ROM (Read only memory), a hard disk, and a memory card.

[0036] The display unit 7 receives and displays the output data of the learning model 43, i.e., data related to the posture estimation of the worker 60, via the communication unit 5. Examples of the display unit 7 include a display dedicated to display and a display including a touch panel capable of detecting touch by a human finger, etc.

[0037] FIG. 4 is a flowchart illustrating a learning model generating method according to this embodiment. First, in step S1, the imaging unit 62 attached to the head of the worker 60 captures an image from the head of the worker 60 toward the front of the worker 60, and obtains work data 41 relating to the work.

[0038] Fig. 2 shows a first specific example of task data 41 captured and acquired by the imaging unit 62. In the first specific example shown in Fig. 2, the worker 60 faces forward. In this case, the imaging unit 62 captures an area from the feet to the chest in front of the worker 60 within its imaging range and can acquire the area as task data 41.

[0039] Fig. 3 is a second specific example of the task data 41 captured and acquired by the imaging unit 62. In the second specific example shown in Fig. 3, the worker 60 is facing the head to the left or tilting the head to the left. Even in this case, the imaging unit 62 can capture the area from the navel to the chest in front of the worker 60 within its imaging range and acquire it as task data 41.

[0040] In step S2 following step S1, the task data 41 is stored in the storage unit 4. Then, in step S3, the teacher data generating unit 31 associates the task data 41 with labels indicating the respective key points of the chest, arms, and hands of the worker 60 when the key point of the median plane 65 of the worker 60 is taken as the starting point. Specifically, as shown in FIG. 2 and FIG. 3, the teacher data generating unit 31 assigns labels indicating the key point K1 of the left chest of the worker 60, the key point K2 of the right chest of the worker 60, the key point K3 of the left elbow of the worker 60, the key point K4 of the right elbow of the worker 60, the key point K5 of the left wrist of the worker 60, and the key point K6 of the right wrist of the worker 60 when the key point K0 of the navel of the worker 60 is taken as the starting point to the task data 41. In step S3, the teacher data generation unit 31 may associate the task data 41 with labels indicating key points of the feet of the worker 60 when the key points of the median plane 65 of the worker 60 are taken as the starting point.

[0041] In step S3, the teacher data generation unit 31 associates the activity data 41 with a label indicating a bounding box of the hand of the worker 60. Specifically, as shown in Fig. 2, the teacher data generation unit 31 attaches a label indicating a bounding box B1 of the left hand of the worker 60 and a bounding box B2 of the right hand of the worker 60 to the activity data 41.

[0042] In step S3, the teacher data generation unit 31 associates the activity data 41 with a label indicating a bounding box of the feet of the worker 60. Specifically, as shown in Fig. 2, the teacher data generation unit 31 attaches a label indicating a bounding box B3 of the left foot of the worker 60 and a bounding box B4 of the right foot of the worker 60 to the activity data 41. The teacher data generation unit 31 generates teacher data 42 by performing such annotation.

[0043] In addition, in step S3, the teacher data generation unit 31 does not necessarily have to associate the work data 41 with labels indicating the respective key points of the chest and arms of the worker 60 when the key point of the median plane 65 of the worker 60 is used as the starting point, and does not necessarily have to associate the work data 41 with labels indicating the respective bounding boxes of the hands and feet of the worker 60.

[0044] In step S4 following step S3, the teacher data 42 is stored in the storage unit 4. Then, in step S5, the machine learning unit 32 performs machine learning using the teacher data 42 to generate a learning model 43 that receives the task data 41 acquired by the imaging unit 62 as an input and outputs data related to posture estimation of the worker 60. Then, in step S6, the learning model 43 is stored in the storage unit 4.

[0045] According to this embodiment, the imaging unit 62 is attached to the head of the worker 60, captures an image from the head of the worker 60 toward the front of the worker 60, and acquires task data 41 related to the task in front of the worker 60. Then, the teacher data generation unit 31 associates the task data 41 as input data acquired by the imaging unit 62 with labels indicating key points (K5, K6) of the hand of the worker 60 when the key point (K0) of the median plane 65 of the worker 60 is taken as the starting point. In addition, the teacher data generation unit 31 associates the task data 41 as input data acquired by the imaging unit 62 with labels indicating respective key points (K1 to K4) of the chest and the arm of the worker 60 when the key point (K0) of the median plane 65 of the worker 60 is taken as the starting point, and associates the task data 41 with respective bounding boxes (B1 to B4) of the hands and feet of the worker 60. As a result, the teacher data generating unit 31 generates the teacher data 42. Therefore, regardless of the place where the worker 60 performs the work and the direction and posture of the body of the worker 60, the teacher data generating unit 31 can execute correspondence between the key points (K1 to K6) of each part of the worker 60 when the key point (K0) of the median plane 65 of the worker 60 is set as the starting point and the work data 41, and correspondence between the bounding boxes (B1 to B4) of each part of the worker 60 and the work data 41. As a result, the machine learning unit 32 can generate a learning model 43 that uses the work data 41 acquired by the imaging unit 62 as input and outputs data related to the posture estimation of the worker 60, regardless of the place where the worker 60 performs the work and the direction and posture of the body of the worker 60, such as the head.

[0046] Moreover, the imaging unit 62 is attached to the forehead of the worker 60, captures an image from the forehead of the worker 60 toward the front of the worker 60, and acquires task data related to the task in front of the worker 60, so that the area in front of the worker 60 can be captured more reliably within the imaging range and acquired as task data 41. Furthermore, the teacher data generation unit 31 associates the key points (K1 to K6) of each part of the worker 60 with the task data 41 when the navel key point K0 on the median plane 65 of the worker 60 is taken as the starting point. This allows the machine learning unit 32 to more reliably generate a learning model 43 that uses the task data 41 acquired by the imaging unit 62 as input and outputs data related to the estimated posture of the worker 60, regardless of the location where the worker 60 performs the task and the orientation and posture of the body such as the head of the worker 60.

[0047] Next, a second embodiment of the present invention will be described. In addition, in cases where the components of the work analysis device according to the second embodiment are similar to the components of the work analysis device according to the first embodiment described above with reference to Figures 1 to 4, duplicate explanations will be omitted as appropriate, and the following explanation will focus on the differences.

[0048] FIG. 5 is a schematic diagram showing a specific example of work data captured and acquired by the imaging section of the present embodiment. FIG. 6 is a schematic diagram illustrating labels that express the relative distance between keypoints as discrete values ​​in a plurality of divisions.

[0049] The work analysis device 2 according to the second embodiment of the present invention generates a learning model for estimating the actions of a worker and analyzes the work performed by the worker. The main configuration of the work analysis device 2 according to the second embodiment is similar to the main configuration of the work analysis device 2 according to the first embodiment.

[0050] The teacher data generation unit 31 of this embodiment generates teacher data by matching the task data 41 as input data acquired by the imaging unit 62 and stored in the memory unit 4 with labels that represent the relative distances of key points on each part of the worker 60 when a key point on the median plane 65 of the worker 60 is used as the starting point, as discrete values ​​in multiple divisions.

[0051] 5, the teacher data generating unit 31 generates teacher data by associating the task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with labels expressing relative distances of key points on the hand of the worker 60 as a starting point from a key point on the median plane 65 of the worker 60, using discrete values ​​as multiple divisions, and stores the teacher data in the storage unit 4. The key point on the median plane 65 of the worker 60 is specifically the key point K0 of the navel of the worker 60.

[0052] In addition, the teacher data generation unit 31 may generate teacher data by matching the task data 41 as input data acquired by the imaging unit 62 and stored in the memory unit 4 with labels that represent the relative distances between each of the key points on the chest, arms, and feet of the worker 60, when a key point on the median plane 65 of the worker 60 is used as the starting point, as discrete values ​​in multiple divisions, and store the teacher data in the memory unit 4.

[0053] 5, the key points on the hand of the worker 60 include a key point K5 on the left wrist of the worker 60, a key point K6 on the right wrist of the worker 60, a key point K7 on the fingers of the left hand of the worker 60, and a key point K8 on the fingers of the right hand of the worker 60. The key points on the hand of the worker 60 may also include a key point on the palm of the left hand of the worker 60, a key point on the palm of the right hand of the worker 60, a key point on the back of the left hand of the worker 60, and a key point on the back of the right hand of the worker 60.

[0054] The key points of the chest of the worker 60 include a key point K1 of the left chest of the worker 60 and a key point K2 of the right chest of the worker 60. The key points of the arms of the worker 60 include a key point K3 of the left elbow of the worker 60 and a key point K4 of the right elbow of the worker 60. The key points of the feet of the worker 60 include a key point of the left foot of the worker 60 and a key point of the right foot of the worker 60.

[0055] In this embodiment, as shown in Fig. 6, a case will be described as an example in which the teacher data generating unit 31 associates the work data 41 with a label expressing the relative distance of the key point K6 of the right wrist of the worker 60 as a discrete value in a plurality of divisions when the key point K0 of the navel on the median plane 65 of the worker 60 is taken as the starting point. Note that the following description of the association is similar to the case in which the teacher data generating unit 31 associates the work data 41 with a label expressing the relative distance of the key points K1 to K5 and K7 to K8 other than the right wrist of the worker 60 as a discrete value in a plurality of divisions when the key point K0 of the navel on the median plane 65 of the worker 60 is taken as the starting point.

[0056] As shown in FIG. 6, the discrete values ​​for the relative distance of the key points include a first discrete value 66 for the horizontal relative distance in the work data 41 and a second discrete value 67 for the vertical relative distance in the work data 41. The "horizontal direction" in this embodiment is an example of the "first direction" of the present invention. The "vertical direction" in this embodiment is an example of the "second direction" of the present invention. The first direction and the second direction are not limited to being perpendicular to each other, but may intersect each other. In other words, the first direction and the second direction are not parallel to each other, but intersect each other. In the description of this embodiment, a case will be taken as an example in which the "first direction" of the present invention is the horizontal direction and the "second direction" of the present invention is the vertical direction.

[0057] For example, when calculating the first discrete value 66, the teacher data generating unit 31 uses the following formula. X coordinate of keypoint K6 on the right wrist - X coordinate of keypoint K0 on the navel ·· Equation (1)

[0058] When the calculation result (difference) of formula (1) is negative, the teacher data generation unit 31 outputs "1" as the first discrete value 66 for the horizontal relative distance. When the calculation result (difference) of formula (1) is equal to or greater than 0 and less than the first threshold, the teacher data generation unit 31 outputs "2" as the first discrete value 66 for the horizontal relative distance. When the calculation result (difference) of formula (1) is equal to or greater than the first threshold and less than the second threshold, the teacher data generation unit 31 outputs "3" as the first discrete value 66 for the horizontal relative distance. When the calculation result (difference) of formula (1) is not within any of the above ranges, the teacher data generation unit 31 outputs "4" as the first discrete value 66 for the horizontal relative distance. In the example shown in FIG. 6, the teacher data generation unit 31 outputs "2" as the first discrete value 66 for the horizontal relative distance.

[0059] Furthermore, for example, when calculating the second discrete value 67, the teacher data generating unit 31 uses the following formula. Y coordinate of navel keypoint K0 - Y coordinate of right wrist keypoint K6 ···Equation (2)

[0060] When the calculation result (difference) of the formula (2) is negative, the teacher data generation unit 31 outputs "4" as the second discrete value 67 for the relative distance in the vertical direction. When the calculation result (difference) of the formula (2) is equal to or greater than 0 and less than the first threshold, the teacher data generation unit 31 outputs "3" as the second discrete value 67 for the relative distance in the vertical direction. When the calculation result (difference) of the formula (2) is equal to or greater than the first threshold and less than the second threshold, the teacher data generation unit 31 outputs "2" as the second discrete value 67 for the relative distance in the vertical direction. When the calculation result (difference) of the formula (2) is not within any of the above ranges, the teacher data generation unit 31 outputs "1" as the second discrete value 67 for the relative distance in the vertical direction. In the example shown in FIG. 6, the teacher data generation unit 31 outputs "4" as the second discrete value 67 for the relative distance in the vertical direction.

[0061] The calculation formula used by the teacher data generating unit 31 to calculate the first discrete value 66 is not limited to the above-mentioned formula (1). The calculation formula used by the teacher data generating unit 31 to calculate the second discrete value 67 is not limited to the above-mentioned formula (2). Furthermore, the number of discrete values ​​(i.e., the number of divisions) regarding the relative distance of the keypoint is not limited to the four values ​​"1", "2", "3" and "4". For example, the teacher data generating unit 31 may automatically set the number of discrete values ​​(i.e., the number of divisions) regarding the relative distance of the keypoint according to the task data 41 acquired by the imaging unit 62 and stored in the storage unit 4.

[0062] 5, the teacher data generation unit 31 generates teacher data by associating the task data 41 as input data acquired by the imaging unit 62 and stored in the storage unit 4 with labels expressing the area of ​​the bounding box of the hand of the worker 60 as discrete values ​​in multiple divisions, and stores the teacher data in the storage unit 4. As shown in FIG. 5, the bounding box of the hand of the worker 60 includes a bounding box B1 of the left hand of the worker 60 and a bounding box B2 of the right hand of the worker 60.

[0063] Moreover, the teacher data generating unit 31 generates teacher data by associating the task data 41 as input data acquired by the imaging unit 62 and stored in the memory unit 4 with labels expressing the area of ​​the bounding box of the feet of the worker 60 as discrete values ​​in multiple divisions, and stores the teacher data in the memory unit 4. As shown in Fig. 5, the bounding box of the feet of the worker 60 includes a bounding box B3 of the left foot of the worker 60 and a bounding box B4 of the right foot of the worker 60.

[0064] For example, if the area of ​​the bounding box is equal to or greater than 0 and less than a first threshold, the teacher data generation unit 31 outputs "1" as the discrete value for the area of ​​the bounding box. If the area of ​​the bounding box is equal to or greater than the first threshold and less than a second threshold, the teacher data generation unit 31 outputs "2" as the discrete value for the area of ​​the bounding box. If the area of ​​the bounding box is equal to or greater than the second threshold and less than a third threshold, the teacher data generation unit 31 outputs "3" as the discrete value for the area of ​​the bounding box. If the area of ​​the bounding box is not within any of the above ranges, the teacher data generation unit 31 outputs "4" as the discrete value for the area of ​​the bounding box.

[0065] It should be noted that the number of discrete values ​​for the area of ​​the bounding box (i.e., the number of sections) is not limited to the four previously mentioned "1," "2," "3," and "4." For example, the teacher data generating unit 31 may automatically set the number of discrete values ​​for the area of ​​the bounding box (i.e., the number of sections) in accordance with the task data 41 acquired by the imaging unit 62 and stored in the memory unit 4.

[0066] The machine learning unit 32 performs machine learning using the teacher data 42 stored in the memory unit 4, receives the task data 41 acquired by the imaging unit 62 as an input, generates a learning model 43 that outputs data related to an estimated movement of the worker 60, and stores the model in the memory unit 4. Specifically, the machine learning unit 32 generates the learning model 43 that outputs data related to an estimated movement of the worker 60 based on time-series changes in the discrete values ​​related to the relative distances of the key points and the discrete values ​​related to the area of ​​the bounding box.

[0067] In this way, the machine learning unit 32 performs machine learning using the teacher data 42 stored in the storage unit 4, generates a learning model 43 for estimating the movement of the worker 60, and stores it in the storage unit 4. The machine learning unit 32 uses the teacher data 42 stored in the storage unit 4 to generate the learning model 43 by a known algorithm such as a neural network (NN). As a result, when task data 41 is input as input data, a learning model 43 is generated that outputs data related to the movement estimation of the worker 60.

[0068] The display unit 7 receives and displays the output data of the learning model 43, i.e., data related to the motion estimation of the worker 60, via the communication unit 5. Examples of the display unit 7 include a display dedicated to display and a display including a touch panel capable of detecting touch by a human finger, etc. The other configuration is the same as that of the work analysis device 2 according to the first embodiment described above with reference to FIGS.

[0069] FIG. 7 is a flowchart illustrating a learning model generating method according to this embodiment. FIG. 8 is a table showing an example of labels in which the relative distance of each keypoint is expressed as a discrete value in a plurality of intervals. FIG. 9 is a table showing an example of labels in which the area of ​​each bounding box is expressed as discrete values ​​in a number of segments.

[0070] First, steps S11 to S12 are the same as steps S1 to S2 described above with reference to Fig. 4. In step S13 following step S12, the teacher data generating unit 31 associates the task data 41 with a label expressing the relative distance of the key points of the hand of the worker 60 as a discrete value as a plurality of divisions when the key point of the median plane 65 of the worker 60 is taken as the starting point. Specifically, the teacher data generating unit 31 assigns to the task data 41 a label expressing the relative distance of each of the key points K5 of the left wrist of the worker 60, the key point K6 of the right wrist of the worker 60, the key point K7 of the fingers of the left hand of the worker 60, and the key point K8 of the fingers of the right hand of the worker 60 as a discrete value as a plurality of divisions when the key point K0 of the navel of the worker 60 is taken as the starting point.

[0071] At this time, the teacher data generating unit 31 may attach to the work data 41 a label expressing the respective relative distances between the key point K1 on the left chest of the worker 60, the key point K2 on the right chest of the worker 60, the key point K3 on the left elbow of the worker 60, and the key point K4 on the right elbow of the worker 60, as discrete values ​​in a plurality of divisions, when the key point K0 on the navel of the worker 60 is used as the starting point. The teacher data generating unit 31 may also attach to the work data 41 a label expressing the respective relative distances between the key point on the left foot of the worker 60 and the key point on the right foot of the worker 60, as discrete values ​​in a plurality of divisions, when the key point K0 on the navel of the worker 60 is used as the starting point.

[0072] An example in which the teacher data generation unit 31 has added labels to the work data 41, representing the relative distances of the key points K1 to K8 of each part of the worker 60, when the key point K0 of the navel of the worker 60 is taken as the starting point, as discrete values ​​in multiple divisions, is as shown in Figure 8.

[0073] In step S13, the teacher data generation unit 31 associates the task data 41 with a label expressing the area of ​​the bounding box of the hand of the worker 60 as a discrete value with multiple divisions. Specifically, the teacher data generation unit 31 attaches to the task data 41 a label expressing the area of ​​each of the bounding box B1 of the left hand of the worker 60 and the bounding box B2 of the right hand of the worker 60 as a discrete value with multiple divisions when the key point K0 of the navel of the worker 60 is set as the starting point.

[0074] In step S13, the teacher data generation unit 31 associates the task data 41 with a label expressing the area of ​​the bounding box of the foot of the worker 60 as a discrete value having a plurality of divisions. Specifically, the teacher data generation unit 31 attaches to the task data 41 a label expressing the area of ​​each of the bounding box B3 of the left foot of the worker 60 and the bounding box B4 of the right foot of the worker 60 as a discrete value having a plurality of divisions when the key point K0 of the navel of the worker 60 is set as the starting point.

[0075] An example in which the teacher data generation unit 31 has added labels to the work data 41, representing the areas of the bounding boxes B1 to B4 of each part of the worker 60, when the key point K0 of the navel of the worker 60 is taken as the starting point, as discrete values ​​in multiple divisions, is shown in Figure 9.

[0076] In addition, in step S13, the teacher data generation unit 31 does not necessarily have to associate the task data 41 with labels that represent the areas of the bounding boxes of the hands of the worker 60 and the feet of the worker 60 as discrete values ​​divided into multiple divisions.

[0077] The teacher data generation unit 31 generates teacher data 42 by performing such annotation.

[0078] Step S14 following step S13 is the same as step S4 described above with reference to Fig. 4. In step S15 following step S14, the machine learning unit 32 performs machine learning using the teacher data 42, and generates a learning model 43 that receives the task data 41 acquired by the imaging unit 62 as an input and outputs data related to the motion estimation of the worker 60. Specifically, the machine learning unit 32 generates a learning model 43 that outputs data related to the motion estimation of the worker 60 based on time-series changes in each discrete value related to the relative distance of the key points and each discrete value related to the area of ​​the bounding box. Next, step S16 following step S15 is the same as step S16 described above with reference to Fig. 4.

[0079] According to this embodiment, the imaging unit 62 captures an image in front of the worker 60, and acquires task data 41 related to the task in front of the worker 60. The teacher data generation unit 31 then generates teacher data 42 by associating the task data 41, which is input data acquired by the imaging unit 62, with labels that express the relative distances of the key points (K5 to K8) on the worker's hand, when the key point (K0) on the median plane of the worker 60 is used as the starting point, as discrete values ​​in a plurality of divisions. In this way, the teacher data generation unit 31 generates the teacher data 42 by calculating the actual relative distance of each key point (a specific distance, for example, 25.5 cm) and the actual area of ​​each bounding box (for example, 225.5 cm). 2 The teacher data 42 is generated by associating the labels with the relative distance of each keypoint and the area of ​​each bounding box expressed as a discrete value in a plurality of divisions (four divisions in this embodiment) rather than a specific area such as a boundary area. This makes it possible to prevent the labels from becoming too large and reduce the number of times that the input data is associated with the labels, i.e., annotation. This simplifies the annotation, i.e. prevents annotation from becoming complicated and difficult, and allows the generation of a learning model 43 that can efficiently estimate the actions of the worker 60 when performing work.

[0080] Furthermore, the machine learning unit 32 performs machine learning using teacher data 42 generated based on task data 41 related to the previous task of the worker 60 captured and acquired by the imaging unit 62, rather than teacher data generated based on data related to targets (objects) such as tools and installations. This makes it possible to prevent labels from becoming too large and reduce the number of times input data is associated with labels, i.e., annotation. This simplifies annotation, i.e., prevents annotation from becoming complicated and difficult, and makes it possible to generate a learning model 43 that can efficiently estimate the actions of the worker 60 when performing a task.

[0081] The teacher data generating unit 31 also associates the relative distances of the key points (K1 to K8) of each part of the worker 60, when the navel key point K0 on the median plane 65 of the worker 60 is set as the starting point, with labels that are expressed as discrete values ​​in multiple divisions. This allows the teacher data generating unit 31 to perform annotation with higher accuracy and in fewer iterations to generate teacher data 42, regardless of the location where the worker 60 works and the orientation and posture of the body of the worker 60, such as the head.

[0082] Furthermore, when the teacher data generation unit 31 automatically sets the number of discrete values ​​(i.e., the number of divisions) for the relative distance of each key point and the area of ​​each bounding box according to the work data 41, it is possible to reduce the number of annotations and simplify annotation while flexibly responding to the content of the work performed by the worker 60.

[0083] The above describes the embodiments of the present invention. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the claims. The configurations of the above embodiments can be partially omitted or arbitrarily combined in a manner different from the above. [Explanation of symbols]

[0084] 2: Work analysis device, 3: Control unit, 4: Memory unit, 5: Communication unit, 7: Display unit, 21: Computer, 31: Teacher data generation unit, 32: Machine learning unit, 41: Work data, 42: Teacher data, 43: Learning model, 44: Program, 60: Worker, 61: Worker terminal, 62: Imaging unit, 63: Helmet, 64: Cable, 65: Median plane, 66: First discrete value, 67: Second discrete value

Claims

1. 1. A learning model generation method for generating a learning model used in an activity analysis device that analyzes activities performed by humans, comprising: a step of capturing an image from the head toward the front of the person using an imaging unit attached to the person's head to acquire work data related to the work; A step of generating training data by associating the acquired task data with labels indicating key points of the person's hands when a key point on the median face of the person is used as a starting point; performing machine learning using the teacher data to generate a learning model in which the acquired task data is input and data related to the person's posture estimation is output; A learning model generation method comprising:

2. The method for generating a learning model according to claim 1 , wherein the key point of the median plane is a key point of the navel of the person.

3. The method of claim 1 , wherein the hand keypoints include a left wrist keypoint of the person and a right wrist keypoint of the person.

4. The method for generating a learning model according to claim 1 , wherein the hand keypoints include finger keypoints of the person's left hand and finger keypoints of the person's right hand.

5. The method for generating a learning model according to claim 1 , wherein the hand keypoints include keypoints on a palm of the person's left hand and keypoints on a palm of the person's right hand.

6. The method for generating a learning model according to claim 1 , wherein the hand keypoints include keypoints on the back of the person's left hand and keypoints on the back of the person's right hand.

7. The learning model generating method according to claim 1, characterized in that the step of generating training data further comprises associating the acquired work data with labels indicating key points of the human chest.

8. The method for generating a learning model according to claim 7 , wherein the chest keypoints include a left chest keypoint of the person and a right chest keypoint of the person.

9. The learning model generation method according to claim 1, characterized in that the step of generating training data further comprises matching the acquired task data with labels indicating key points of the person's arm.

10. The method for generating a learning model according to claim 9 , wherein the arm keypoints include a keypoint of a left elbow of the person and a keypoint of a right elbow of the person.

11. The learning model generating method according to claim 1, characterized in that the step of generating training data further comprises associating the acquired task data with labels indicating key points of the person's feet.

12. The method for generating a learning model according to claim 11, wherein the foot keypoints include a keypoint of a left foot of the person and a keypoint of a right foot of the person.

13. The learning model generating method according to claim 1 , wherein the step of generating the training data further comprises associating the acquired task data with a label indicating the area of ​​the person's hand.

14. The method for generating a learning model according to claim 13 , wherein the hand regions include a region of the person's left hand and a region of the person's right hand.

15. The learning model generating method according to claim 1 , wherein the step of generating the training data further comprises associating the acquired task data with a label indicating the area of ​​the person's foot.

16. The method for generating a learning model according to claim 15 , wherein the foot region includes a left foot region of the person and a right foot region of the person.

17. The learning model generation method according to claim 1, characterized in that the step of acquiring the work data comprises acquiring the work data by capturing an image from the forehead toward the front of the person using the imaging unit attached to the person's forehead.

18. The learning model generation method according to claim 17, characterized in that the imaging unit is a wearable camera attached to either the brim of a helmet, the brim of a hat, or a belt worn by the person on the head.

19. A task analysis device for analyzing tasks performed by people, comprising: an imaging unit that is attached to the person's head and captures an image from the head toward the front of the person to acquire work data related to the work; a teacher data generation unit that generates teacher data by associating the task data acquired by the imaging unit with labels indicating key points of the person's hand when a key point on the median face of the person is used as a starting point; a machine learning unit that performs machine learning using the teacher data to generate a learning model in which the task data acquired by the imaging unit is input and data related to the posture estimation of the person is output; A work analysis device comprising:

20. A task analysis program executed by a computer of a task analysis device that analyzes tasks performed by people, The computer includes: a step of capturing an image from the head toward the front of the person using an imaging unit attached to the person's head to acquire work data related to the work; A step of generating training data by associating the acquired task data with labels indicating key points of the person's hands when a key point on the median face of the person is used as a starting point; performing machine learning using the teacher data to generate a learning model in which the acquired task data is input and data related to the person's posture estimation is output; A work analysis program characterized by executing the above.

Citation Information

Patent Citations

  • Clothing matching method and device, electronic equipment and storage medium

    CN116012619A

  • Work similarity calculation method, device and program

    JP2016066270A

  • Image processing device, image processing method, and image processing program

    JP2016099643A

  • Behavior recognition device, behaviour learning device, method, and program

    JP2016099982A

  • Image evaluation device, image evaluation method, and computer program

    JP2019175321A