Work analysis device, work analysis method, and program

Abstract

To automate the identification of the type of work performed by an operator.SOLUTION: A work analysis device comprises: a video acquisition unit which acquires work video in which a work site is captured; a facility position information acquisition unit which acquires facility position information indicative of the position of the facility related to the work included in the work video; a region extraction unit which extracts a region image, which is a region including the image of the worker, from the acquired work video; a skeleton model extraction unit which extracts a skeleton model corresponding to the worker's posture from the extracted region image; a work type determination unit which determines the type of work corresponding to the worker's posture indicated by the extracted skeleton model and the position of the facility related to the work indicated by the facility position information, on the basis of a pre-trained learning model which has learned the relationship among the worker's posture, the position of the facility related to the work, and the type of work performed by the worker; and an output unit which outputs the determination results of the work type.SELECTED DRAWING: Figure 18

Description

[Technical field]

[0001] The present disclosure relates to an activity analysis device, an activity analysis method, and a program. [Background technology]

[0002] When it comes to work analysis, such as instantaneous observation (work sampling) or continuous observation, which have traditionally been done manually, it has been proposed to automate the process by using video from a fixed camera (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-206321 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, because workers vary widely in terms of their physiques and clothing, it is difficult to automatically identify movements and postures using feature values ​​alone.

[0005] The present disclosure has been made in consideration of these circumstances, and aims to provide a work analysis device, a work analysis method, and a program that can automate the determination of a worker's work type and improve the accuracy of the determination. [Means for solving the problem]

[0006] One aspect of the present disclosure is a work analysis device that includes a video acquisition unit that acquires a work video in which a work site is captured, an equipment location information acquisition unit that acquires equipment location information indicating the location of equipment related to the work included in the work video, an area extraction unit that extracts an area image from the acquired work video that is an area including an image of a worker, a skeletal model extraction unit that extracts a skeletal model corresponding to the posture of the worker from the extracted area image, a work type determination unit that determines the type of work corresponding to the posture of the worker indicated by the extracted skeletal model and the position of the equipment related to the work indicated by the equipment location information based on a learning model in which a correspondence between the posture of the worker, the position of equipment related to the work, and the type of work performed by the worker has been learned in advance, and an output unit that outputs the determination result of the type of work.

[0007] One aspect of the present disclosure is a work analysis method including a computer device comprising: a video acquisition unit that acquires a work video in which a work area is captured; acquiring equipment position information indicating the position of equipment related to the work included in the work video; extracting an area image from the acquired work video that is an area including an image of a worker; extracting a skeletal model corresponding to the posture of the worker from the extracted area image; determining a type of work corresponding to the posture of the worker indicated by the extracted skeletal model and the position of the equipment related to the work indicated by the equipment position information based on a learning model in which a correspondence between the posture of the worker, the position of equipment related to the work, and the type of work performed by the worker has been learned in advance; and outputting a determination result of the type of work.

[0008] One aspect of the present disclosure is a program for causing a computer to perform the following operations: acquire a work video in which a work area is captured; acquire equipment location information indicating the location of equipment related to the work included in the work video; extract an area image from the acquired work video which is an area including an image of a worker; extract a skeletal model corresponding to the posture of the worker from the extracted area image; determine the type of work corresponding to the posture of the worker indicated by the extracted skeletal model and the position of equipment related to the work indicated by the equipment location information based on a learning model in which the correspondence between the posture of the worker, the position of equipment related to the work, and the type of work performed by the worker has been learned in advance; and output the determination result of the type of work. Effect of the Invention

[0009] According to the task analysis device, task analysis method, and program disclosed herein, it is possible to automate the determination of a worker's task type and improve the accuracy of the determination. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of an overview of an operation analysis system according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a diagram showing an example of a work video. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning information generating device according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of operation class information according to the present embodiment. [Diagram 5] A diagram showing an overview of the combination of information at the learning stage of the learning model of this embodiment. [Figure 6] 11 is a diagram showing an example of association between area images and action classes according to the present embodiment; FIG. [Figure 7] FIG. 2 is a diagram showing an example of a processing flow of the learning information generating device of the present embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a functional configuration of the work analysis device according to the present embodiment. [Figure 9]FIG. 11 is a diagram showing an example of an operation determination result according to the present embodiment. [Figure 10] 11 is a diagram illustrating an example of a method for determining an activity class by an activity type determination unit of the present embodiment. FIG. [Figure 11] FIG. 4 is a diagram showing an example of a processing flow of the work analysis device of the present embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of a functional configuration of a learning information generating device according to a second embodiment. [Figure 13] FIG. 2 is a diagram showing an example of a position of a machine tool according to the present embodiment. [Figure 14] FIG. 4 is a diagram illustrating an example of facility location information according to the present embodiment. [Figure 15] A diagram showing an overview of the combination of information at the learning stage of the learning model of this embodiment. [Figure 16] FIG. 2 is a diagram showing an example of a processing flow of the learning information generating device of the present embodiment. [Figure 17] FIG. 2 is a diagram illustrating an example of a functional configuration of the work analysis device according to the present embodiment. [Figure 18] FIG. 2 is a diagram showing an example of a flow of the work analysis device of the present embodiment. [Figure 19] FIG. 13 is a diagram illustrating a modified example of the configuration of the work analysis system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [Overview of learning information generating device 1] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. 1 is a diagram showing an example of an overview of a work analysis system 100 according to this embodiment. The work analysis system 100 analyzes the work status of a worker P who operates a machine tool 90, for example, on a production line in a factory. The analysis of the work status in this embodiment includes, for example, identifying the type of work performed by the worker P by a so-called instantaneous observation method (work sampling method) or continuous observation method. The types of work include, for example, equipment adjustment, walking, inspection, monitoring, work collection, etc. In the following description, the types of work are also referred to as work classes.

[0012] The work analysis system 100 of this embodiment includes a learning information generation device 1 and an work analysis device 2 (or a learning information generation device 10 and an work analysis device 20), and an imaging device 3. The imaging device 3 captures the work scene of the worker P and generates a work video 30. The imaging device 3 outputs the generated work video 30 to the learning information generation device 1 and the work analysis device 2 (or the learning information generation device 10 and the work analysis device 20). The imaging device 3 may output the work video 30 to the learning information generation device 1 and the work analysis device 2 (or the learning information generation device 10 and the work analysis device 20) via a storage medium (not shown) having a semiconductor memory or a storage device (not shown) having a hard disk or the like (for example, non-real time). The imaging device 3 may also output the work video 30 to the learning information generation device 1 and the work analysis device 2 (or the learning information generation device 10 and the work analysis device 20) directly (for example, in real time) without going through a storage medium (not shown) or a storage device (not shown).

[0013] In the following explanation, when it is particularly necessary to indicate a direction or position, an xyz three-axis Cartesian coordinate system will be used. The z-axis indicates the vertical upward direction. The x-axis and y-axis indicate a plane parallel to the factory floor. The x-axis indicates the horizontal direction of the work video 30, and the y-axis indicates the depth direction of the work video 30.

[0014] The imaging device 3 is installed so that the production line on which multiple machine tools 90 are arranged is included in the imaging range 31. For example, the imaging device 3 is installed in a position where it can overlook the work passage of the production line and where it does not interfere with the movement of the worker P. The work video 30 captured by the imaging device 3 may include multiple machine tools 90 and multiple workers P. In one example of the figure, the imaging axis AX of the imaging device 3 is directed toward the manufacturing line in which the machine tools 90 are arranged (for example, the direction of the passageway along which the worker P moves, that is, the direction of the work flow line).

[0015] The worker P may be in charge of a plurality of machine tools 90, that is, a so-called multi-machine owner, or may be in charge of one machine tool 90 exclusively. When multiple workers P are included in the image capture range 31, each worker P may wear an accessory for identifying the worker P. In one example in the same figure, the workers P wear work caps C of different colors as an accessory for identifying the worker P. In addition, the wearable item may be anything that can identify each worker P, and may be a work hat C, work clothes, a name tag, shoes, a sanitary mask, protective glasses, etc., or may be a tool with a distinctive appearance.

[0016] 2 is a diagram showing an example of a work video 30. At least eight machine tools 90, from machine tool 90-1 to machine tool 90-8, are included in the imaging range 31 of the imaging device 3. An image obtained by cutting out (extracting) an area including an image of a worker P from this work video 30 is also referred to as an area image 11. Note that an image of the worker P is omitted from the drawing.

[0017] The work analysis system 100 of this embodiment determines the type of work performed by worker P using a learning model 70 that has learned the posture of worker P and the situation around worker P (e.g., the positional relationship between worker P and the machine tool 90) for each type of work.

[0018] The learning information generating device 1 and the learning information generating device 10 are devices that generate information (learning information 60) for learning the learning model 70 in the learning stage of the learning model 70. The work analysis device 2 and the work analysis device 20 are devices that determine the work type of the worker P using the learning model 70 at the stage of utilizing the learning model 70. The learning information generation device 1 and the work analysis device 2, and the learning information generation device 10 and the work analysis device 20 use some different information for learning and judgment (inference). In the first embodiment, a learning information generating device 1 and an work analysis device 2 will be described below. In the second embodiment, a learning information generating device 10 and an work analysis device 20 will be described.

[0019] [First embodiment] The work analysis system 100 includes a learning information generation device 1 and an work analysis device 2. In the following description, the learning information generation device 1 and the work analysis device 2 are described as separate computers, but this is not limited thereto. The learning information generation device 1 and the work analysis device 2 may be included in the same computer.

[0020] [Functional configuration of learning information generation device 1] 3 is a diagram showing an example of a functional configuration of the learning information generation device 1 of the first embodiment. The learning information generation device 1 is connected to a display device 4, an operation device 5, and a learning device 6 in addition to the imaging device 3 described above.

[0021] The display device 4 includes, for example, a liquid crystal display, and presents the information output from the learning information generation device 1 to the user of the learning information generation device 1. The operation device 5 includes, for example, a keyboard, a mouse, a touch panel, and the like, detects an operation by a user of the learning information generation device 1, and outputs to the learning information generation device 1 information indicating the detected operation.

[0022] The learning information generation device 1 includes a calculation unit 110, a storage unit 120, a video acquisition unit 130, and an output unit 150. The calculation unit 110 is a computer device and includes a central processing unit (CPU). The calculation unit 110 provides various functions based on the programs and data stored in the storage unit 120 . The storage unit 120 includes a semiconductor memory and a hard disk device, and stores programs and data for the operation of the calculation unit 110 .

[0023] The video acquisition unit 130 acquires the work video 30 output by the imaging device 3. As described above, the imaging device 3 captures an image of a work place such as a production line in a factory, and outputs the work video 30. That is, the video acquisition unit 130 acquires the work video 30 in which the work place is captured. The video acquisition unit 130 outputs the acquired work video 30 to the calculation unit 110.

[0024] The calculation unit 110 includes, as its functional units, a region extraction unit 111, a skeleton model extraction unit 112, and a learning information generation unit 114.

[0025] The area extraction unit 111 extracts an area image 11 that is an area including an image of the worker P from the acquired work video 30. The area extraction unit 111 outputs the extracted area image 11 to the skeletal model extraction unit 112 and the display device 4. The skeletal model extraction unit 112 extracts a skeletal model 12 of the worker P from the extracted area image 11. The skeletal model 12 is information indicating the posture of the worker P based on the lengths of bones and the positions and angles of joints. The skeletal model extraction unit 112 estimates the skeleton of the worker P by a known method based on an image of the appearance of the worker P included in the extracted area image 11. For example, the skeletal model extraction unit 112 identifies the head position, neck position, shoulder position, elbow position, wrist position, fingertip position, waist position, knee position, ankle position, toe position, and the like from the appearance of the worker P included in the area image 11. The skeletal model extraction unit 112 determines the posture of the worker P based on the relative positional relationship of each of the identified positions of the worker P. The skeletal model extraction unit 112 estimates the skeleton corresponding to the posture of the worker P based on the determined posture of the worker P and a standard skeletal model stored in advance. For example, the skeletal model extraction unit 112 determines the positions of joints such as shoulder joints, elbow joints, and wrist joints from the posture of the worker P. The skeletal model extraction unit 112 estimates the skeleton corresponding to the posture of the worker P by increasing or decreasing the length of each bone of the standard skeletal model according to the position of each joint determined. In this manner, the skeletal model extraction unit 112 extracts the skeletal model 12 corresponding to the posture of the worker P from the area image 11.

[0026] The learning information generator 114 generates learning information 60 for learning the learning model 70, based on the skeletal model 12 and the movement class information 50 output from the operation device 5. The movement class information 50 is information indicating the type of work performed by the worker P. An example of the movement class information 50 will be described with reference to FIG.

[0027] 4 is a diagram showing an example of the action class information 50 of this embodiment. The action class information 50 is information in which an action class (equipment adjustment, walking, inspection, monitoring, work collection, etc.), which is a type of work, is associated with an action class ID that identifies the action class. For example, the action class "device adjustment" is associated with the action class ID "ID01."

[0028] 5 is a diagram showing an overview of a combination of information at the learning stage of the learning model 70 of this embodiment. The skeletal model 12 is information indicating the posture of the worker P based on the lengths of bones and the positions and angles of joints.

[0029] 6 is a diagram showing an example of association between the skeleton model 12 and the action class in this embodiment. The skeleton model 12 is output from the skeleton model extraction unit 112 to the display device 4, and is displayed on the display device 4. A user of the learning information generating device 1 looks at the skeletal model 12 displayed on the display device 4 and determines the action class indicated by the skeletal model 12. Note that the display device 4 may display, in addition to the skeletal model 12, the area image 11 from which the skeletal model 12 is extracted. Also, the display device 4 may display the area image 11 from which the skeletal model 12 is extracted, without displaying the skeletal model 12. According to the learning information generation device 1 configured in this manner, when it is difficult to determine the movement class using only the skeletal model 12, it is possible to determine the movement class while observing the situation around the worker P shown in the area image 11. A user of the learning information generation device 1 inputs the determined action class to the operation device 5. The operation device 5 outputs the action class specified by the user to the calculation unit 110 as action class information 50. This action class information 50 functions as teacher information when generating a learning model 70.

[0030] 5, the learning information generation unit 114 generates learning information 60 that associates the skeletal model 12 with the motion class information 50. The learning information generation unit 114 outputs the generated learning information 60 to the output unit 150. The output unit 150 outputs the learning information 60 to the learning device 6 .

[0031] The learning device 6 is a computer device that uses a known machine learning algorithm to generate a learning model 70 based on the correspondence between the skeletal model 12 and the action class information 50. The learning model 70 thus trained has a performance of outputting the type of action (action class) of the worker P included in the image when the skeletal model 12 is given. In the following description, the learning model 70 that is generated based on the correspondence between the skeleton model 12 and the action class information 50 as described above will also be referred to as a learning model 71.

[0032] The learning device 6 stores the generated learning model 71 in the learning model storage unit 7. The learning model storage unit 7 is, for example, an external server device of the learning information generating device 1 or a virtual device such as a cloud server, and includes a semiconductor memory and a hard disk device. The learning model storage unit 7 stores the learning model 71 generated by the learning device 6.

[0033] In the present embodiment, the learning device 6 and the learning model storage unit 7 are described as devices external to the learning information generation device 1, but this is not limited thereto. The learning information generation device 1 may also have the functions of the learning device 6 and the learning model storage unit 7.

[0034] [Learning process flow] The process flow of each of the above-mentioned functions will be described with reference to FIG. FIG. 7 is a diagram showing an example of a processing flow of the learning information generating device 1 of the present embodiment. (Step S110) The video acquisition unit 130 acquires the work video 30 from the imaging device 3. (Step S120) The area extraction unit 111 extracts the area image 11 from the work video 30 acquired by the video acquisition unit . (Step S 130 ) The skeletal model extraction unit 112 extracts the skeletal model 12 from the region image 11 . (Step S140) The learning information generating unit 114 acquires the action class information 50 from the operation device 5. As described above, this action class information 50 is used as teacher information when generating the learning model 70. (Step S150) The learning information generator 114 generates learning information 60 by associating the skeleton model 12 with the action class information 50. (Step S160) The output unit 150 outputs the learning information 60 to the learning device 6. (Step S170) The learning device 6 generates the learning model 70 (learning model 71). The learning device 6 stores the generated learning model 70 (learning model 71) in the learning model storage unit 7.

[0035] [Functional configuration of work analysis device 2] Next, the functional configuration of the work analysis device 2 of this embodiment will be described. 8 is a diagram showing an example of the functional configuration of the work analysis apparatus 2 of this embodiment. The work analysis apparatus 2 is connected to a display device 4 and a learning model storage unit 7, in addition to the imaging device 3 described above. The same functions and configurations as those described in the above-mentioned learning information generation device 1 are denoted by the same reference numerals, and the description thereof will be omitted.

[0036] The work analysis device 2 includes a calculation unit 210, a storage unit 220, a video acquisition unit 230, and an output unit 250. The computing unit 210 is a computer device and includes a central processing unit (CPU). The calculation unit 210 provides various functions based on the programs and data stored in the storage unit 220 . The storage unit 220 includes a semiconductor memory and a hard disk device, and stores programs and data for the operation of the calculation unit 210 .

[0037] The video acquisition unit 230 acquires the work video 30 output by the imaging device 3. As described above, the imaging device 3 captures an image of a work place such as a production line in a factory, and outputs the work video 30. That is, the video acquisition unit 230 acquires the work video 30 in which the work place is captured. The video acquisition unit 230 outputs the acquired work video 30 to the calculation unit 210.

[0038] The calculation unit 210 includes, as its functional units, a region extraction unit 211, a skeleton model extraction unit 212, a recognition unit 214, and an activity type determination unit 215.

[0039] The area extraction unit 211 extracts an area image 11, which is an area including an image of the worker P, from the acquired work video 30. The area extraction unit 211 outputs the extracted area image 11 to the skeletal model extraction unit 212 and the identification unit 214.

[0040] The skeletal model extraction unit 212 extracts a skeletal model 12 of the worker P from the extracted area image 11. The skeletal model extraction unit 212 outputs the extracted skeletal model 12 to the recognition unit 214 and the task type determination unit 215. Note that the skeletal model extraction unit 212 extracts the skeletal model 12 of the worker P from the area image 11 by a known method, similar to the above-mentioned skeletal model extraction unit. The identification unit 214 uses the skeleton model 12 to identify the worker P included in the area image 11 , and outputs the identification result 14 to the task type determination unit 215 .

[0041] As described above, the skeletal model 12 is an image of a skeleton corresponding to the posture of the worker P. In one example of this embodiment, the posture of the worker P is a posture represented by a human skeletal model. The skeletal model extraction unit 212 extracts a skeletal model corresponding to the posture of the worker P as the skeletal model 12.

[0042] That is, the skeletal model extraction unit 212 extracts the skeletal model 12 corresponding to the posture of the worker P from the extracted area image 11.

[0043] More specifically, the skeletal model extraction unit 212 cuts out an image of the appearance of the worker P from the area image 11. The skeletal model extraction unit 212 estimates a skeletal model of the worker P (such as bone lengths, joint angles, and joint positions) corresponding to the appearance of the worker P from the cut-out image of the worker P's appearance based on a known skeletal estimation means. The skeletal model extraction unit 212 extracts the estimated skeletal model of the worker P as the skeletal model 12.

[0044] The skeletal model extraction unit 212 configured in this way extracts an estimated result of the skeleton, not the appearance of the worker P, as the skeletal model 12, so that the judgment can be made without being influenced by variations in the appearance of the worker P (for example, sagging or wrinkles in the work clothes). Furthermore, the skeletal model extraction unit 212 configured in this way can reduce concerns about leakage of personal information and resistance of the worker P when an image of the worker P's appearance is used as is for analysis.

[0045] The identification unit 214 identifies the worker P based on the feature amount of the worker P's clothing included in the area image 11. In one example of this embodiment, the workers P in the imaging range 31 are wearing work hats C of different colors. The identification unit 214 identifies the colors of the work hats C of the workers P, and identifies the workers P with different colors of work hats C as different workers P. Furthermore, the identification unit 214 identifies the workers P with the same color of work hat C included in different times (frames) of the work video 30 as the same worker P. The identification unit 214 can also track the movement of a plurality of workers P on a time axis while identifying the workers P based on the feature amount of the worker P's clothing (for example, the color of the work cap C). The identification unit 214 outputs the worker ID of the worker P (for example, the first worker P1) to the task type determination unit 215 as the worker identification result 14.

[0046] That is, the identification unit 214 identifies the worker P included in the work video 30. If the identification unit 214 identifies the worker P and the work video 30 includes images of multiple workers P, the region extraction unit 211 extracts a region image 11 for each worker P. Furthermore, when the work video 30 includes images of multiple workers P, the skeletal model extraction unit 212 extracts a skeletal model 12 corresponding to the posture of each worker P from the extracted area image 11.

[0047] If it is clear that the work video 30 includes only one worker P, there is no need to identify each of the multiple workers P. In this case, the work analysis device 2 does not need to include the identification unit 214.

[0048] In addition, in one example of this embodiment, it has been described that the skeletal model 12 extracted by the skeletal model extraction unit 212 includes multiple workers P, and the identification unit 214 identifies each of the workers P included in the area image 11 (or the work video 30), but this is not limited to this. The identification unit 214 may be configured to use the work video 30 as an image to be identified, identify the worker P included in the work video 30, and output the identification result to the skeletal model extraction unit 212. In this case, when images of multiple workers P are included in the work video 30, the skeletal model extraction unit 212 extracts area images 11 for each of the multiple workers, the skeletal model extraction unit 212 extracts skeletal models 12 for each of the multiple workers P from each area image 11, the identification unit 214 identifies the multiple workers based on the area images 11 for each of the multiple workers, and the work type determination unit 215 determines the type of work for each worker P. That is, in this case, the skeletal model 12 is generated for each worker P.

[0049] The work type determination unit 215 determines the type of work corresponding to the surrounding environment of worker P indicated by the posture of worker P shown by the extracted skeletal model 12, based on a learning model 70 in which the correspondence between the posture of worker P, the surrounding environment of worker P, and the type of work performed by worker P is pre-learned.

[0050] Here, the learning model storage unit 7 stores the learning model 70 (learning model 71) learned in the above-mentioned learning stage. The task type determination unit 215 reads out the learning model 70 stored in the learning model storage unit 7, and generates the task determination result 25 by applying the skeleton model 12 to the read-out learning model 70.

[0051] In this embodiment, the task type determination unit 215 generates the task determination result 25 by reading out the learning model 70, but this is not limited to the above. For example, there may be an external determination device (not shown) that can refer to the learning model 70. In this case, the task type determination unit 215 may generate the task determination result 25 by supplying the skeleton model 12 and the worker identification result 14 to the external determination device and acquiring the determination result (i.e., the task determination result 25) from this external determination device.

[0052] The task type determination section 215 outputs the determination result to the output section 250 as the task determination result 25 .

[0053] 9 is a diagram showing an example of the task determination result 25 of this embodiment. The task type determination unit 215 associates the worker ID identified by the identification unit 214 with the time of the action and the task class, and outputs the result as the task determination result 25.

[0054] 10 is a diagram showing an example of a method for determining a work class by the work type determination unit 215 of this embodiment. The work type determination unit 215 determines the work class of a worker P captured in the work video 30 by a predetermined first sampling period T1 (short-time step). The task type determining section 215 determines a representative class for each predetermined second sampling period T2 (long time step) for the task type determined for each first sampling period T1 (short time step). As an example, the first sampling period T1 (short time step) is 1 second (once per second), and the second sampling period T2 (long time step) is 20 seconds (3 times per minute). In this case, the work type determination unit 215 determines the work class for each first sampling period T1 (1 second), and determines the work class that appears most frequently within the second sampling period T2 (20 seconds) (in this figure, "ID02: walking") as the representative class for the second sampling period T2. That is, the task type determining unit 215 determines a task class in a first sampling period T1, and determines a representative class that represents the task class in a second sampling period T2 that is longer than the first sampling period T1.

[0055] The above-mentioned first sampling period T1 (short time step) and second sampling period T2 (long time step) are merely examples and are not limited thereto. For example, the first sampling period T1 (short time step) may be 1 / 60 to 1 / 30 seconds (30 to 60 times per second), and the second sampling period T2 (long time step) may be 1 to 2 minutes.

[0056] Returning to Fig. 8, the output section 250 outputs the work judgment result 25 judged by the work type judgment section 215 (for example, the work judgment result 25 shown in Fig. 9) to the display device 4. As a result, the work judgment result 25 is displayed on the display device 4.

[0057] [Processing flow at the utilization stage] The process flow of each of the above-mentioned functions will be described with reference to FIG. FIG. 11 is a diagram showing an example of the processing flow of the work analysis device 2 of this embodiment.

[0058] (Step S210) The video acquisition unit 230 acquires the work video 30 from the imaging device 3. The video acquisition unit 230 can acquire the work video 30 in real time from the imaging device 3, and can also acquire the work video 30 captured in the past by the imaging device 3 via a video storage device (not shown). In other words, the work analysis device 2 can analyze the work currently being performed by worker P on the production line in real time (i.e., online analysis), and can also analyze work performed in the past at a later date (i.e., offline analysis).

[0059] (Step S220) The area extraction unit 211 extracts the area image 11 from the work video 30 acquired by the video acquisition unit 230. (Step S230) The skeletal model extraction unit 212 extracts the skeletal model 12 from the region image 11 extracted in step S220. (Step S240) The identification unit 214 identifies the worker P included in the area image 11 (or the work video 30). The identification unit 214 outputs the identification result to the work type determination unit 215 as the worker identification result 14. (Step S250) The work type determination unit 215 determines the type of work corresponding to the surrounding environment of the worker P indicated by the extracted skeletal model 12, based on a learning model 70 in which the correspondence between the posture of the worker P, the surrounding environment of the worker P, and the type of work performed by the worker P is pre-learned. Note that the above-mentioned steps S240 and S250 may be executed in parallel after the completion of step S230 as shown in the figure. (Step S260) The output unit 250 outputs the result of the determination made in step S250 (task determination result 25) to the display device 4.

[0060] As described above, the work analysis system 100 of this embodiment infers work classes based on work videos 30 (i.e., fixed-point videos) captured over a wide area of ​​a factory production line, etc., using an imaging device 3 installed at a fixed point, and a learning model that has been trained on work types. According to the work analysis system 100 of this embodiment, it is possible to automate the determination of the work type of the worker P by the instantaneous observation method (work sampling method) or the continuous observation method. Therefore, compared to the conventional method of manual determination, it is possible to observe the work for a longer period of time and perform the analysis in a shorter time.

[0061] Furthermore, according to the work analysis system 100 of this embodiment, a plurality of machine tools 90 and a plurality of workers P can be accommodated within the imaging range 31 and observed simultaneously. Therefore, according to the work analysis system 100 of this embodiment, work performed simultaneously in parallel at a plurality of locations on the production line can also be analyzed.

[0062] Furthermore, the work analysis system 100 of this embodiment can identify (for example, track) the worker P. Therefore, according to the work analysis system 100 of this embodiment, even if the worker P goes outside the imaging range 31 or the movement lines of multiple workers P overlap, it is possible to analyze each worker P.

[0063] [Second embodiment] Next, a learning information generation device 10 and a work analysis device 20 of a second embodiment will be described. The learning information generation device 10 and the work analysis device 20 of this embodiment differ from the learning information generation device 1 and the work analysis device 2 of the first embodiment described above in that information indicating the positions of each part of a machine tool 90 arranged in a production line (equipment position information 80) is used in the learning stage and the utilization stage. It should be noted that parts having the same functions and configurations as those of the above-described learning information generation device 1 and work analysis device 2 are given the same reference numerals, and descriptions thereof will be omitted unless otherwise specified.

[0064] [Functional configuration of learning information generation device 10] FIG. 12 is a diagram illustrating an example of a functional configuration of the learning information generating device 10 according to the second embodiment. The learning information generation device 10 includes a calculation unit 110, a storage unit 120, a video acquisition unit 130, a facility location information acquisition unit 140, and an output unit 150. The configurations of the storage unit 120, the video acquisition unit 130, and the output unit 150 are similar to those of the above-described learning information generation device 1, and therefore a description thereof will be omitted.

[0065] The equipment location information acquisition unit 140 acquires equipment location information 80 from the equipment information storage unit 8. The equipment location information 80 is information that indicates the location of equipment related to the work included in the work video 30.

[0066] 13 is a diagram showing an example of the position of a machine tool 90 in this embodiment. As shown in the figure, the machine tool 90 is captured in a work video 30 captured by an imaging device 3. The machine tool 90 has, for example, an operation panel 91, a machine tool main body 92, a workpiece removal port 93, and other components. In addition, a work position 94 for a worker P is located on the passage in front of the machine tool 90. The facility position information 80 is generated by extracting the positions of the operation panel 91, the machine tool body 92, the workpiece removal port 93, the work position 94, etc., from an image (for example, a certain frame of the work video 30) captured by the imaging device 3 using a known image processing method (or by selecting based on a user's operation on the image). When there are multiple imaging devices 3, the facility position information 80 is generated for each imaging device 3. The facility position information 80 is information determined based on the relative positional relationship between the imaging device 3 and the machine tool 90. The facility position information 80 indicates which pixel range, among the pixels constituting the work video 30, each position related to the machine tool 90 (for example, the operation panel 91, the machine tool body 92, the workpiece removal port 93, and the work position 94) corresponds to.

[0067] Fig. 14 is a diagram showing an example of facility location information 80 in this embodiment. In this example, facility location information 80 indicates the positions of each part of machine tool 90 in work video 30 shown in Fig. 13. For example, facility location information 80 indicates operation panel 91 of machine tool 90-5, machine tool body 92, workpiece removal port 93, and work position 94 when work is performed on machine tool 90-5. Note that in this figure, the reference numerals for machine tools 90-1 to -4 and machine tools 90-6 to -8 are omitted. The work position 94 refers to a basic work position when working on the machine tool 90-5, or a position of equipment related to the work.

[0068] Returning to FIG. 12, the facility location information acquisition unit 140 outputs the acquired facility location information 80 to the learning information generation unit 114.

[0069] The calculation unit 110 includes, as its functional units, a region extraction unit 111, a skeleton model extraction unit 112, and a learning information generation unit 114. The configurations of the region extraction unit 111 and the skeleton model extraction unit 112 are similar to those of the above-mentioned learning information generation device 1, and therefore a description thereof will be omitted.

[0070] The learning information generation unit 114 generates learning information 61 for learning the learning model 70 based on the skeleton model 12, the equipment position information 80, and the action class information 50. In the following description, the learning model 70 generated based on the correspondence between the skeleton model 12, the equipment position information 80, and the action class information 50 will also be referred to as a learning model 72.

[0071] 15 is a diagram showing an overview of a combination of information in the learning stage of the learning model 72 of this embodiment. The learning information generation unit 114 generates learning information 61 that associates the skeleton model 12, the equipment position information 80, and the operation class information 50. The learning information generation unit 114 outputs the generated learning information 61 to the output unit 150. The output unit 150 outputs the learning information 61 to the learning device 6 .

[0072] The learning device 6 generates a learning model 72 based on the correspondence between the skeleton model 12, the equipment position information 80, and the action class information 50. Learning model 72 learns the positions of each part of machine tool 90 indicated by facility position information 80, including the corresponding relationships with other information. As a result, even if the position of the machine tool 90 captured in the work video 30 has changed, the learning model 72 can estimate the action class by providing the equipment position information 80 after the position has changed. That is, the learning model 72 thus trained has the ability to output the type of work (action class) of the worker P included in the image when the skeletal model 12 and the equipment position information 80 are provided.

[0073] [Learning process flow] The process flow of each of the above-mentioned functions will be described with reference to FIG. FIG. 16 is a diagram showing an example of a processing flow of the learning information generating device 10 of the present embodiment. (Step S310) The video acquisition unit 130 acquires the work video 30 from the imaging device 3. (Step S320) The facility location information acquisition unit 140 acquires the facility location information 80 from the facility information storage unit 8. (Step S330) The area extraction unit 111 extracts the area image 11 from the work moving image 30 acquired by the moving image acquisition unit . (Step S340) The skeletal model extraction unit 112 extracts the skeletal model 12 from the region image 11. (Step S350) The learning information generating unit 114 acquires the action class information 50 from the operation device 5. The action class information 50 is used as teacher information when generating the learning model 72. (Step S360) The learning information generator 114 generates learning information 61 by associating the skeleton model 12, the equipment position information 80, and the action class information 50 with each other. (Step S370) The output unit 150 outputs the learning information 61 to the learning device 6. (Step S380) The learning device 6 generates the learning model 72. The learning device 6 stores the generated learning model 72 in the learning model storage unit 7.

[0074] [Functional configuration of the work analysis device 20] Next, the functional configuration of the work analysis device 20 of this embodiment will be described. FIG. 17 is a diagram showing an example of the functional configuration of the work analysis device 20 of this embodiment.

[0075] The work analysis device 20 includes a calculation unit 210, a storage unit 220, a video acquisition unit 230, a facility location information acquisition unit 240, and an output unit 250.

[0076] The video acquisition unit 230 acquires the work video 30 output by the imaging device 3. As described above, the imaging device 3 captures an image of a work place such as a production line in a factory, and outputs the work video 30. That is, the video acquisition unit 230 acquires the work video 30 in which the work place is captured. The video acquisition unit 230 outputs the acquired work video 30 to the calculation unit 210.

[0077] The facility location information acquisition unit 240 acquires facility location information 81 from the facility information storage unit 8. The facility location information 81 is information corresponding to the facility location information 80 (see, for example, FIG. 14) stored in the facility information storage unit 8 in the learning stage. Here, the equipment location information 81 may be the equipment location information 80 itself stored in the equipment information memory unit 8 during the learning stage, or the position of the machine tool 90 in the work video 30 may be information different from the equipment location information 80. More specifically, the equipment location information 81 only needs to indicate which pixel range, among the pixels constituting the work video 30, each position on the machine tool 90, such as the operation panel 91, the machine tool main body 92, the workpiece removal port 93, and the work position 94, corresponds to. In other words, the equipment location information 81 may be the equipment location information 80 itself, or may be information different from the equipment location information 80. That is, the equipment location information acquisition section 240 acquires information indicating the location of equipment related to the work included in the work video 30 (equipment location information 81).

[0078] The calculation unit 210 includes, as its functional units, a region extraction unit 211, a skeletal model extraction unit 212, a recognition unit 214, and an operation type determination unit 215.

[0079] The area extraction unit 211 extracts an area image 11, which is an area including an image of the worker P, from the acquired work video 30. The area extraction unit 211 outputs the extracted area image 11 to the skeletal model extraction unit 212 and the identification unit 214.

[0080] The skeletal model extraction unit 212 extracts a skeletal model 12 of the worker P from the extracted area image 11. The skeletal model extraction unit 212 outputs the extracted skeletal model 12 to the task type determination unit 215.

[0081] As described above, the skeletal model 12 is an image that represents the posture of the worker P by the positions and angles of the joints. That is, the skeletal model extraction unit 212 extracts a skeletal model that corresponds to the posture of the worker P as the skeletal model 12.

[0082] More specifically, the skeletal model extraction unit 212 cuts out an image of the appearance of the worker P from the area image 11. The skeletal model extraction unit 212 estimates a skeletal model of the worker P (such as bone lengths, joint angles, and joint positions) corresponding to the appearance of the worker P from the cut-out image of the worker P's appearance based on a known skeletal estimation means. The skeletal model extraction unit 212 extracts the estimated skeletal model of the worker P as the skeletal model 12.

[0083] The skeletal model extraction unit 212 configured in this way extracts the skeletal model 12 based on the positions and angles of the joints rather than the appearance of the worker P, so that the judgment can be made without being influenced by the variation in the appearance of the worker P (for example, sagging or wrinkles in the work clothes). Furthermore, the skeletal model extraction unit 212 configured in this way can reduce concerns about the leakage of personal information when an image of the appearance of the worker P is used directly for analysis, and can reduce the resistance of the worker P.

[0084] The identification unit 214 identifies the worker P based on the feature amount of the worker P's clothing included in the skeletal model 12. In one example of this embodiment, the workers P in the imaging range 31 are wearing work hats C of different colors. The identification unit 214 identifies the colors of the work hats C of the workers P, and identifies the workers P with different colors of work hats C as different workers P. Furthermore, the identification unit 214 identifies the workers P with the same color of work hat C included in different times (frames) of the work video 30 as the same worker P. The identification unit 214 can also track the movement of a plurality of workers P on a time axis while identifying the workers P based on the feature amount of the worker P's clothing (for example, the color of the work cap C). The identification unit 214 outputs the worker ID of the worker P (for example, the first worker P1) to the task type determination unit 215 as the worker identification result 14.

[0085] If it is clear that the work video 30 includes only one worker P, there is no need to identify each of the multiple workers P. In this case, the work analysis apparatus 20 does not need to include the identification unit 214.

[0086] In addition, in one example of this embodiment, it has been described that the skeletal model 12 extracted by the skeletal model extraction unit 212 includes multiple workers P, and the identification unit 214 identifies each of the workers P included in the skeletal model 12, but this is not limited to this. The identification unit 214 may be configured to use the work video 30 as an image to be identified, identify the worker P included in the work video 30, and output the identification result to the skeletal model extraction unit 212. In this case, when images of multiple workers P are included in the work video 30, the skeletal model extraction unit 212 extracts area images 11 for each of the multiple workers, the skeletal model extraction unit 212 extracts skeletal models 12 for each of the multiple workers P from each area image 11, the identification unit 214 identifies the workers based on the area images 11 for each of the multiple workers, and the work type determination unit 215 determines the type of work for each worker P. That is, in this case, the skeletal model 12 is generated for each worker P.

[0087] The work type determination unit 215 determines the type of work corresponding to the posture of the worker P indicated by the extracted skeletal model 12 and the position of the equipment related to the work indicated by the equipment position information 81, based on a learning model 72 in which the correspondence between the posture of the worker P, images of the worker's surroundings and the position of equipment related to the work, and the type of work performed by the worker P has been pre-learned.

[0088] Here, the learning model 72 learned in the above-mentioned learning stage is stored in the learning model storage unit 7. The task type determination unit 215 reads out the learning model 72 stored in the learning model storage unit 7, and generates a task determination result 25 by providing the learning model 72 read out with the skeletal model 12 corresponding to the extracted posture of the worker P in correspondence with the equipment position information 81.

[0089] [Processing flow at the utilization stage] The process flow of each of the above-mentioned functions will be described with reference to FIG. FIG. 18 is a diagram showing an example of a flow of the work analysis device 20 of this embodiment.

[0090] (Step S410) The video acquisition unit 230 acquires the work video 30 from the imaging device 3. The video acquisition unit 230 can acquire the work video 30 in real time from the imaging device 3, and can also acquire the work video 30 captured in the past by the imaging device 3 via a video storage device (not shown). In other words, the work analysis device 2 can analyze the work currently being performed by worker P on the production line in real time (i.e., online analysis), and can also analyze work performed in the past at a later date (i.e., offline analysis).

[0091] (Step S420) The facility location information acquisition unit 240 acquires the facility location information 81 from the facility information storage unit 8. (Step S430) The area extraction unit 211 extracts the area image 11 from the work moving image 30 acquired by the moving image acquisition unit 230. (Step S440) The skeletal model extraction unit 212 extracts the skeletal model 12 from the region image 11 extracted in step S430. (Step S450) The identification unit 214 identifies the worker P included in the area image 11 (or the work video 30). The identification unit 214 outputs the identification result to the work type determination unit 215 as the worker identification result 14. (Step S460) The work type determination unit 215 determines the type of work corresponding to the posture of worker P indicated by the extracted skeletal model 12 and the position of equipment related to the work indicated by the equipment position information 81, based on a learning model 70 in which the correspondence between the posture of worker P, the environment surrounding worker P and the position of equipment related to the work, and the type of work performed by worker P has been learned in advance. Note that the above-mentioned steps S450 and S460 may be executed in parallel after the completion of step S440 as shown in the figure. (Step S470) The output unit 250 outputs the result of the determination made in step S250 (task determination result 25) to the display device 4.

[0092] As described above, the work analysis system 100 of this embodiment infers work classes based on work videos 30 (i.e., fixed-point videos) captured over a wide area of ​​a factory production line, etc., using an imaging device 3 installed at a fixed point, and a learning model that has been trained on work types. According to the work analysis system 100 of this embodiment, it is possible to automate the determination of the work type of the worker P by the instantaneous observation method (work sampling method) or the continuous observation method. Therefore, compared to the conventional method of manual determination, it is possible to observe the work for a longer period of time and perform the analysis in a shorter time.

[0093] Furthermore, according to the work analysis system 100 of this embodiment, a plurality of machine tools 90 and a plurality of workers P can be accommodated within the imaging range 31 and observed simultaneously. Therefore, according to the work analysis system 100 of this embodiment, work performed simultaneously in parallel at a plurality of locations on the production line can also be analyzed.

[0094] Furthermore, the work analysis system 100 of this embodiment can identify (for example, track) the worker P. Therefore, according to the work analysis system 100 of this embodiment, even if the worker P goes outside the imaging range 31 or the movement lines of multiple workers P overlap, it is possible to analyze each worker P.

[0095] [Variations] 19 is a diagram showing a modified example of the configuration of the work analysis system 100. In each of the above-mentioned embodiments, the work analysis system 100 is described as having one imaging device 3, but this is not limited to the above. As shown in the diagram, the work analysis system 100 may have multiple imaging devices 3.

[0096] In this modified example, the video acquisition section 230 acquires work videos 30 captured in a plurality of work locations. The region extraction unit 211 extracts region images 11 from each of the acquired work videos 30. The skeleton model extraction unit 212 extracts a skeleton model 12 for each of the extracted region images 11. The task type determining unit 215 determines the type of task for each of the skeleton models 12. The output unit 250 outputs, as the work determination result 25, the results of tallying up the work type work determination results 25 based on the multiple work videos 30 for each worker P based on the identification result of the worker P.

[0097] In this case, each of the work videos 30 captured by the imaging devices 3-1 to -4 is assigned a time code. The video acquisition section 130 aligns the time axes of the work videos 30 acquired from the multiple image capture devices 3 based on the time codes. The calculation unit 110 tracks the worker P in the multiple work videos 30 whose time axes are aligned, thereby preventing duplicate counting when the same worker P is included in multiple image capture ranges 31. When the same worker P is included in multiple imaging ranges 31, the results of determining the type of work may differ among the multiple work videos 30. In this case, the calculation unit 110 may set the representative class estimated by statistical processing such as majority vote or mode selection of the determination results based on the multiple work videos 30 as the work determination result 25.

[0098] According to the work analysis system 100 configured in this manner, by integrating the work videos 30 captured by the multiple imaging devices 3, the system can be adapted to cases where the production line does not fit within the imaging range 31 of one imaging device 3.

[0099] Furthermore, when the calculation unit 110 generates the work judgment result 25 for the work video 30 provided from the imaging device 3, the calculation unit 110 may discard the no longer needed work video 30 without storing it. According to the work analysis system 100 configured in this manner, it is possible to prevent the storage of a huge amount of information about the work video 30.

[0100] A program for implementing the functions of any of the components in any of the devices described above may be recorded on a computer-readable recording medium, and the program may be read into a computer system for execution. Note that the term "computer system" as used herein includes an operating system or hardware such as peripheral devices. Also, the term "computer-readable recording medium" may include a flexible disk, a magneto-optical disk, a ROM, a CD (Compact Disc)-ROM (Read-Only Memory), etc. The term "computer-readable recording medium" refers to a portable medium such as a portable only memory (PDA), and a storage device such as a hard disk built into a computer system. Furthermore, "computer-readable recording medium" also includes a device that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The volatile memory may be, for example, a RAM (Random Access Memory). The recording medium may be, for example, a non-transitory recording medium.

[0101] The above-mentioned program may be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has a function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. The above program may be for implementing some of the above functions. Furthermore, the above program may be a so-called differential file that can implement the above functions in combination with a program already recorded in the computer system. The differential file may be called a differential program.

[0102] In addition, the function of any of the components in any of the above-described devices may be realized by a processor. For example, each process in the embodiment may be realized by a processor that operates based on information such as a program and a computer-readable recording medium that stores information such as a program. Here, the processor may, for example, realize the functions of each unit with individual hardware, or may realize the functions of each unit with integrated hardware. For example, the processor may include hardware, and the hardware may include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor may be configured using one or more circuit devices mounted on a circuit board, or one or both of one or more circuit elements. An IC (Integrated Circuit) or the like may be used as the circuit device, and a resistor or a capacitor or the like may be used as the circuit element.

[0103] Here, the processor may be, for example, a CPU. However, the processor is not limited to a CPU, and various types of processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) may be used. In addition, the processor may be, for example, an ASIC (Application Specific Integrated Circuit). The processor may be a hardware circuit such as an ASIC (Application Specific Integrated Circuit). The processor may be, for example, configured with multiple CPUs, or may be configured with a hardware circuit such as an ASIC. The processor may be, for example, configured with a combination of multiple CPUs and a hardware circuit such as an ASIC. The processor may include, for example, one or more of an amplifier circuit or a filter circuit that processes an analog signal.

[0104] Although the embodiments of this disclosure have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of this disclosure are also included. [Explanation of symbols]

[0105] 1, 10... learning information generating device, 2, 20... work analysis device, 100... work analysis system 110: calculation unit, 111: area extraction unit, 112: skeleton model extraction unit, 114: learning information generation unit, 120: storage unit, 130: video acquisition unit, 140: equipment position information acquisition unit, 150: output unit 210: calculation unit, 211: area extraction unit, 212: skeleton model extraction unit, 214: identification unit, 215: work type determination unit, 220: storage unit, 230: video acquisition unit, 240: equipment position information acquisition unit, 250: output unit 3...imaging device, 4...display device, 5...operation device, 50...action class information, 6...learning device, 7...learning model storage unit, 8...equipment information storage unit, 90...machine tool, 91...operation panel, 92...machine tool body, 93...work removal port, 94...standard work position, 11...area image, 12...skeletal model, 14...worker identification result, 25...work judgment result, 30...work video, 31...imaging range, 60, 61...learning information, 70, 71, 72...learning model, 711...background learning model, 712...worker learning model, 80...equipment position information, AX...imaging axis, C...work cap, P...worker

Claims

1. A video acquisition unit that acquires a work video in which a work place is captured; an equipment position information acquisition unit that acquires equipment position information indicating a position of equipment related to the work included in the work video; an area extraction unit that extracts an area image that includes an image of a worker from the acquired work video; a skeleton model extraction unit that extracts a skeleton model corresponding to a posture of the worker from the extracted area image; a task type determination unit that determines a task type corresponding to the posture of the worker indicated by the extracted skeletal model and the position of the equipment related to the task indicated by the equipment position information, based on a learning model in which a correspondence relationship between the posture of the worker, the position of the equipment related to the task, and the type of task performed by the worker has been learned in advance; an output unit that outputs the determination result of the type of work; A work analysis device comprising:

2. The work type determination unit determines the type of the work by providing the learning model with a skeleton model corresponding to the extracted posture of the worker in association with the equipment position information. The work analysis apparatus according to claim 1 .

3. An identification unit for identifying a worker included in the work video Further equipped with the region extraction unit extracts the region image for each worker based on a result of identifying the worker when images of a plurality of workers are included in the work video; The skeletal model extraction unit extracts the skeletal model corresponding to a posture of each worker from the extracted area images when images of a plurality of workers are included in the work video. The work analysis apparatus according to claim 1 .

4. The identification unit identifies the worker based on a feature amount of the worker's clothing. The work analysis apparatus according to claim 3 .

5. The video acquisition unit acquires work videos captured in a plurality of work locations, The region extraction unit extracts the region images from each of the plurality of work videos obtained, the skeletal model extraction unit extracts the skeletal model for each of the extracted region images; The work type determination unit determines a type of the work for each pair of the skeleton model and the equipment position information, The output unit outputs, as the determination result, a result of tallying up the determination results of the type of work based on the plurality of work videos for each worker based on the identification result of the worker. The work analysis apparatus according to claim 3 .

6. A computer device comprising: A video acquisition unit that acquires a work video in which a work place is captured; Obtaining equipment position information indicating a position of equipment related to the work included in the work video; Extracting an area image, which is an area including an image of a worker, from the acquired work video; extracting a skeleton model corresponding to a posture of the worker from the extracted region image; determining a type of work corresponding to the posture of the worker indicated by the extracted skeletal model and the position of the equipment related to the work indicated by the equipment position information, based on a learning model in which a correspondence relationship between the posture of the worker, the position of the equipment related to the work, and the type of work performed by the worker has been learned in advance; outputting a result of the determination of the type of work; Work analysis methods, including:

7. On the computer, Acquiring a work video in which a work place is captured; Obtaining equipment position information indicating a position of equipment related to the work included in the work video; Extracting an area image, which is an area including an image of a worker, from the acquired work video; extracting a skeleton model corresponding to a posture of the worker from the extracted region image; determining a type of work corresponding to the posture of the worker indicated by the extracted skeletal model and the position of the equipment related to the work indicated by the equipment position information, based on a learning model in which a correspondence relationship between the posture of the worker, the position of the equipment related to the work, and the type of work performed by the worker has been learned in advance; outputting a result of the determination of the type of work; A program for executing the above.

Images