Action determination device, action determination method, and computer-readable storage medium

Abstract

In an action determination device (10), an image acquisition unit (21) acquires image data about a subject. A skeleton extraction unit (22) extracts object information from the image data acquired by the image acquisition unit (21), the object information being skeleton information representing the posture of the subject such as coordinates of a plurality of joints. An action information registration unit (23) determines the action content shown in the action information as the action content being performed by the subject, the action information being skeleton information similar to the object information extracted by the skeleton extraction unit (22).

Description

Technical Field

[0001] The present invention relates to a technology for specifying the content of an action of a subject person from image data in which the subject person is photographed. Background Art

[0002] In the industrial field, there is a need for processing such as measuring the cycle time, which is the time it takes for an operator to assemble a product, and analyzing the content of a job in order to detect omissions of a job or unstable work that is not a stable job. Currently, it is mainstream to perform these processes manually. For this purpose, a lot of manpower costs are required, and only a limited range can be processed.

[0003] Patent Document 1 describes the use of a camera and a three-dimensional sensor mounted on a person's head to extract feature quantities of a person's movements and automatically perform movement analysis.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2016-099982 Summary of the invention

[0007] Problems to be solved by the invention

[0008] In Patent Document 1, a camera is worn on a person's head. However, in the industrial field, wearing an unnecessary item on a part of a worker's body during work may hinder the work, and thus is avoided.

[0009] An object of the present invention is to enable processing such as measurement of cycle time and analysis of work contents without requiring a worker to wear an item unnecessary for the work on his body.

[0010] Means for solving problems

[0011] The action determination device of the present invention comprises:

[0012] An image acquisition unit that acquires image data about a subject;

[0013] a skeleton extracting unit that extracts object information, which is skeleton information indicating a posture of the subject, from the image data acquired by the image acquiring unit; and

[0014] The action identification unit identifies action content indicated by action information as action content being performed by the subject, wherein the action information is the skeleton information similar to the object information extracted by the skeleton extraction unit.

[0015] Effects of the Invention

[0016] In the present invention, object information, which is skeletal information representing the posture of the subject, is extracted from the image data, and the action content shown by the action information is determined as the action content being performed by the subject, and the action information is skeletal information similar to the object information. Therefore, the operator does not need to wear items that are not necessary for the work on the body, and can perform processing such as measurement of cycle time and analysis of work content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a structural diagram of the action identification device 10 according to the first embodiment.

[0018] Figure 2 This is a flowchart of the registration process in the first embodiment.

[0019] Figure 3 This is an explanatory diagram of image data according to the first embodiment.

[0020] Figure 4 This is an explanatory diagram of the skeleton information 43 according to the first embodiment.

[0021] Figure 5 This is an explanatory diagram of the registration process according to the first embodiment.

[0022] Figure 6 This is an explanatory diagram of the action information table 31 according to the first embodiment.

[0023] Figure 7 This is a flowchart of the determination process in the first embodiment.

[0024] Figure 8 This is an explanatory diagram of the determination processing of the first embodiment.

[0025] Fig. 9 It is a structural diagram of the action identification device 10 according to the first modification.

[0026] Fig.10 It is a structural diagram of the action identification device 10 according to the third modification.

[0027] Fig.11 It is a structural diagram of the action identification device 10 according to the second embodiment.

[0028] Fig.12 This is a flowchart of the learning process in the second embodiment.

[0029] Fig.13 This is a flowchart of the determination processing of implementation mode 2.

[0030] Fig.14 It is a structural diagram of the action identification device 10 according to the fifth modification.

[0031] (Explanation of Reference Numerals)

[0032] 10: action determination device; 11: processor; 12: memory; 13: storage device; 14: communication interface; 15: electronic circuit; 21: image acquisition unit; 22: skeleton extraction unit; 23: action information registration unit; 24: action determination unit; 25: output unit; 26: learning unit; 31: action information table; 32: learning model; 41: shooting device; 42: person; 43: skeleton information. DETAILED DESCRIPTION

[0033] Implementation Method 1

[0034] ***Structure description***

[0035] Reference Figure 1 The structure of the action identification device 10 according to the first embodiment will be described.

[0036] The action determining device 10 is a computer.

[0037] The action identification device 10 includes hardware such as a processor 11, a memory 12, a storage device 13, and a communication interface 14. The processor 11 is connected to other hardware via a signal line and controls the other hardware.

[0038] The processor 11 is an IC (Integrated Circuit) that performs processing. As a specific example, the processor 11 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit).

[0039] The memory 12 is a storage device for temporarily storing data. As a specific example, the memory 12 is SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory).

[0040] The storage device 13 is a storage device for storing data. As a specific example, the storage device 13 is a hard disk drive (HDD). In addition, the storage device 13 may also be a removable storage medium such as a SD (registered trademark, Secure Digital) memory card, a CF (CompactFlash, registered trademark), a NAND flash memory, a floppy disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD (Digital Versatile Disk).

[0041] The communication interface 14 is an interface for communicating with an external device. As a specific example, the communication interface 14 is a port of Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark, High-Definition Multimedia Interface). In addition, the communication interface 14 may be set separately for each data to be communicated. For example, HDMI (registered trademark) may be set for communicating the image data described later, and USB may be set for communicating the tag information described later.

[0042] The action identification device 10 includes, as functional components, an image acquisition unit 21, a skeleton extraction unit 22, an action information registration unit 23, an action identification unit 24, and an output unit 25. The functions of the functional components of the action identification device 10 are realized by software.

[0043] The storage device 13 stores a program for realizing the functions of each functional component of the action identification device 10. The program is read into the memory 12 by the processor 11 and executed by the processor 11. Thus, the functions of each functional component of the action identification device 10 are realized.

[0044] In addition, the storage device 13 stores an action information table 31 .

[0045] exist Figure 1 In the figure, only one processor 11 is shown. However, there may be multiple processors 11, and the multiple processors 11 may jointly execute programs to realize various functions.

[0046] As a specific example, the action determination device 10 may also include a CPU and a GPU as the processor 11. In this case, as described later, the skeleton extraction unit 22 for performing image processing may be implemented by the GPU, while the remaining image acquisition unit 21, the action information registration unit 23, the action determination unit 24, and the output unit 25 may be implemented by the CPU.

[0047] ***Action description***

[0048] Reference Figures 2 to 8 The operation of the operation specifying device 10 according to the first embodiment will be described.

[0049] The operation of the operation identification device 10 in the first embodiment corresponds to the operation identification method in the first embodiment. In addition, the operation of the operation identification device 10 in the first embodiment corresponds to the processing of the operation identification program in the first embodiment.

[0050] The operation of the operation identification device 10 in the first embodiment includes a registration process and an identification process.

[0051] Reference Figure 2 The registration process according to the first embodiment will be described.

[0052] (Step S11: Image acquisition processing)

[0053] The image acquisition unit 21 acquires, via the communication interface 14, one or more sets of image data of a person 42 performing a target action photographed by the photographing device 41 and tag information indicating the target action. Figure 3 As shown, in the first embodiment, the image data is acquired by imaging device 41 capturing the entire body of person 42 performing the target action from the front of the target person.

[0054] The image acquisition unit 21 writes the acquired set of image data and label information into the memory 12 .

[0055] (Step S12: Skeleton extraction processing)

[0056] The skeleton extraction unit 22 reads the image data acquired in step S11 from the memory 12. The skeleton extraction unit 22 extracts skeleton information 43 representing the posture of the person 42 from the image data as motion information. Figure 4 As shown, in the first embodiment, the skeleton information 43 shows the coordinates of a plurality of joints such as the neck and shoulders of the person 42 or the relative positional relationship of the plurality of joints.

[0057] The skeleton extraction unit 22 writes the extracted motion information into the memory 12 .

[0058] (Step S13: Action Information Registration Process)

[0059] The motion information registering unit 23 reads the motion information extracted in step S12 and the label information of the same group as the image data from which the motion information was extracted from the memory 12. The motion information registering unit 23 associates the read motion information and label information and writes them into the motion information table 31.

[0060] (Step S14: End of determination process)

[0061] The skeleton extraction unit 22 determines whether or not the processing has been completed for all the groups acquired in step S11 .

[0062] The skeleton extraction unit 22 ends the registration process when processing has been completed for all groups. On the other hand, when there is an unprocessed group, the skeleton extraction unit 22 returns the process to step S12 and executes processing for the next group.

[0063] By executing the registration process, a plurality of sets of action information and label information are accumulated in the action information table 31 .

[0064] For example, Figure 5 As shown, in step S11, the image acquisition unit 21 acquires the image data at each moment constituting the image data of a person performing a series of operations and a group of label information indicating the action of the person indicated by the image data at that moment. Then, in step S12, the skeleton extraction unit 22 extracts the action information from the image data of the processing object, and in step S13, the action information registration unit 23 writes the label information and action information of the same group as the image data of the processing object into the action information table 31 in correspondence. Thus, as shown in FIG. Figure 6 As shown in FIG. 1 , regarding actions at each time point in a series of operations, the action information table 31 stores the corresponding action information and label information.

[0065] In step S11, the image acquisition unit 21 may acquire a set of image data at each time and label information indicating the action of the person shown in the image data at that time, with respect to image data at each time constituting image data of a person performing an unstable action that is not normally performed in a series of actions. Thus, with respect to the unstable action, the action information and label information corresponding to the action at each time are accumulated in the action information table 31.

[0066] Reference Figure 7 The determination process of the first embodiment will be described.

[0067] (Step S21: Image acquisition processing)

[0068] The image acquisition unit 21 acquires one or more image data of the subject through the communication interface 14. In the first embodiment, the image data acquired in step S21 is acquired by the imaging device 41 capturing the entire body of the subject from the front, similarly to the image data acquired in step S11.

[0069] The image acquisition unit 21 writes the acquired image data into the memory 12 .

[0070] (Step S22: Skeleton Extraction Process)

[0071] The skeleton extraction unit 22 reads the image data acquired in step S21 from the memory 12. The skeleton extraction unit 22 extracts skeleton information 43 indicating the posture of the subject person from the image data as subject information.

[0072] The skeleton extraction unit 22 writes the extracted object information into the memory 12 .

[0073] (Step S23: Action determination processing)

[0074] The action identification unit 24 identifies the action content indicated by the action information as the action content being performed by the subject, the action information being skeleton information similar to the object information extracted in step S22 .

[0075] Specifically, the action determination unit 24 retrieves action information similar to the object information from the action information table 31. Similarity means that, when the skeleton information 43 shows the coordinates of multiple joints, the Euclidean distance between the coordinates of the same joints in the object information and the action information is short. In addition, when the skeleton information 43 shows the relative positional relationship of multiple joints, it means that the Euclidean distance between the joints shown in the object information and the Euclidean distance between the joints shown in the action information are close. Then, the action determination unit 24 determines the action content shown by the label information corresponding to the action information hit in the retrieval as the action content being performed by the subject.

[0076] For example, the action determination unit 24 calculates the similarity with the object information for all the action information accumulated in the action information table 31. Then, the object determination unit 24 processes the action information with the highest similarity as the action information that was hit in the search. In addition, if there is no action information with a confidence level higher than a threshold, the action determination unit 24 may treat it as the action information that was not hit in the search.

[0077] In addition, when the relative positional relationship between specific joints represents an action, weighting can also be performed so that the difference in Euclidean distance about a specific joint has a large impact on the similarity. That is to say, when the skeleton information 43 shows the coordinates of multiple joints, weighting can also be performed so that the difference in Euclidean distance between the coordinates in the object information and the coordinates in the action information about a specific joint has a large impact on the similarity. In addition, when the skeleton information 43 shows the relative positional relationship between multiple joints, weighting can also be performed so that the difference in Euclidean distance between specific joints has a large impact on the similarity.

[0078] (Step S24: Output processing)

[0079] The output unit 25 outputs the action content determined in step S23 to a display device or the like connected via the communication interface 14. The output unit 25 may output tag information indicating the action content.

[0080] In addition, when there is no action information hit in the search, the output unit 25 outputs information indicating that the action content cannot be determined.

[0081] (Step S25: End of determination process)

[0082] The skeleton extraction unit 22 determines whether or not all the image data acquired in step S21 have been processed.

[0083] The skeleton extraction unit 22 ends the registration process when processing has been completed for all the image data. On the other hand, when there is unprocessed image data, the skeleton extraction unit 22 returns the process to step S22 and executes processing for the next image data.

[0084] For example, Figure 8 As shown, in step S21, the image acquisition unit 21 acquires the image data at each moment constituting the image data of the person performing a series of operations. Then, in step S22, the skeleton extraction unit 22 extracts the object information from the image data of the processing object, and in step S23, the motion information registration unit 23 retrieves the motion information similar to the object information and determines the motion content. In this way, the motion content at each moment in a series of operations can be determined.

[0085] At this time, it is also possible to determine when the target operation starts and when it ends. In addition, it is also possible to determine that the target person has performed an unstable operation in a series of operations.

[0086] **Effects of Implementation Method 1**

[0087] As described above, the action determination device 10 of the first embodiment extracts object information as skeletal information representing the posture of the subject from the image data of the subject photographed from the front, and determines the action content indicated by the action information as skeletal information similar to the object information as the action content being performed by the subject. Therefore, the action determination device 10 of the first embodiment takes the image data including a plurality of image data as input, determines the action content about each image data, and can thereby analyze a series of actions. As a result, the operator does not wear any unnecessary items for the work on his body, and can perform such processing as measuring the cycle time and analyzing the work content.

[0088] ***Other structures***

[0089] <Modification 1>

[0090] In implementation mode 1, if Figure 1 As shown, the action determination device 10 is a single device. However, the action determination device 10 may also be a system including a plurality of devices.

[0091] As a specific example, Fig. 9As shown, the action determination device 10 may also be a system including a registration device having a function related to the registration process and a determination device having a function related to the determination process. In this case, the action information table 31 may be stored in a storage device provided outside the registration device and the determination device, or may be stored in a storage device of any device in the registration device and the determination device.

[0092] In addition, Fig. 9 In the figure, the hardware of the registration device and the determination device is omitted. The registration device and the determination device are provided with a processor, a memory, a storage device, and a communication interface as hardware, similar to the action determination device 10.

[0093] <Modification 2>

[0094] In Embodiment 1, data captured by the imaging device 41 is used as the image data. However, three-dimensional image data obtained by a sensor such as a depth sensor may be used as the image data.

[0095] <Variation 3>

[0096] In the first embodiment, each functional component is implemented by software. However, as a modification example 3, each functional component may be implemented by hardware. Regarding this modification example 3, the differences from the first embodiment will be described.

[0097] Reference Fig.10 The structure of the action identification device 10 according to the third modification will be described.

[0098] When each functional component is realized by hardware, the action determination device 10 includes an electronic circuit 15 instead of the processor 11, the memory 12, and the storage device 13. The electronic circuit 15 is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage device 13.

[0099] As the electronic circuit 15, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA (Gate Array), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array) can be envisioned.

[0100] Each functional component may be implemented by one electronic circuit 15 , or may be implemented in a distributed manner in a plurality of electronic circuits 15 .

[0101] <Variation 4>

[0102] As a fourth modification, a part of each functional component may be realized by hardware, and the other functional components may be realized by software.

[0103] The processor 11, the memory 12, the storage device 13 and the electronic circuit 15 are referred to as a processing circuit. That is, the functions of each functional structural element are realized by the processing circuit.

[0104] Implementation Method 2

[0105] Embodiment 2 is different from Embodiment 1 in that a learning model 32 is generated based on action information and label information, and label information corresponding to object information is determined by the learning model 32. This difference will be described in Embodiment 2, and description of the same points will be omitted.

[0106] ***Structure description***

[0107] Reference Fig.11 The structure of the action identification device 10 according to the second embodiment will be described.

[0108] The action identification device 10 includes a learning unit 26 instead of the action information registration unit 23. Figure 1 In addition, regarding the action determination device 10, the storage device 13 stores the learning model 32 instead of the action information table 31, which is different from Figure 1 The action determination means 10 shown is different.

[0109] ***Action description***

[0110] Reference Figure 12 to Figure 13 The operation of the operation specifying device 10 according to the second embodiment will be described.

[0111] The operation of the operation identification device 10 in the second embodiment corresponds to the operation identification method in the second embodiment. In addition, the operation of the operation identification device 10 in the second embodiment corresponds to the processing of the operation identification program in the second embodiment.

[0112] The operation of the operation identification device 10 in the second embodiment includes a learning process and an identification process.

[0113] Reference Fig.12 The learning process of the second embodiment will be described.

[0114] The processing from step S31 to step S32 is the same as Figure 2 The processing of step S11 to step S12 is the same as that of step S34. Figure 2 The processing of step S14 is the same as that of step S14.

[0115] (Step S33: Learning model generation process)

[0116] The learning unit 26 learns the action information extracted in step S32 and a plurality of groups of label information of the same group as the image data from which the action information is extracted as learning data. Thus, the learning unit 26 generates a learning model 32, which determines action information similar to the inputted skeletal information 43 when the skeletal information 43 is inputted, and outputs label information corresponding to the determined action information. As for the method of learning based on the learning data, an existing machine learning model or the like may be used. The learning unit 26 writes the generated learning model 32 into the storage device 13.

[0117] When the learning model 32 has already been generated, the learning unit 26 updates the learning model 32 by providing the learning data to the generated learning model 32 .

[0118] In addition, in step S31, not only the pair of image data and label information but also only the image data may be input. In this case, the action information is extracted from the image data in step S32, and only the action information is provided as learning data to the learning model 32 in step S33. In this way, a certain learning effect can be obtained even in the absence of label information.

[0119] Reference Fig.13 The determination process of the second embodiment will be described.

[0120] The processing from step S41 to step S42 is the same as Figure 7 The processing of step S21 to step S22 is the same as that of step S44 to step S45. Figure 7 The processing from step S24 to step S25 is the same.

[0121] (Step S43: Action determination processing)

[0122] The action determination unit 24 inputs the object information extracted in step S42 into the learning model 32 stored in the storage device 13, and obtains the label information output from the learning model 32. Then, the action determination unit 24 determines the action content indicated by the obtained label information as the action content being performed by the subject. In other words, the action determination unit 24 determines the action content indicated by the label information inferred and output from the object information by the learning model 32 as the action content being performed by the subject.

[0123] **Effects of Implementation Method 2**

[0124] As described above, the action identification device 10 according to the second embodiment generates the learning model 32, and identifies the label information corresponding to the object information using the learning model 32. Therefore, the identification of the label information corresponding to the object information can be performed efficiently.

[0125] ***Other structures***

[0126] <Variation 5>

[0127] In implementation mode 2, Fig.11 As shown, the action identification device 10 is a single device. However, similarly to the first modification, the action identification device 10 may be a system including a plurality of devices.

[0128] As a specific example, Fig.14 As shown, the action determination device 10 may also be a system including a learning device having a function related to the learning process and a determination device having a function related to the determination process. In this case, the learning model 32 may be stored in a storage device provided outside the learning device and the determination device, or may be stored in a storage device of any device in the learning device and the determination device.

[0129] In addition, Fig.14 The hardware of the learning device and the determination device is omitted. The learning device and the determination device are provided with a processor, a memory, a storage device, and a communication interface as hardware, similar to the action determination device 10.

[0130] <Variation 6>

[0131] In implementation mode 2, Fig.11 As shown, the action determination device 10 includes a processor 11, a memory 12, a storage device 13, and a communication interface 14 as hardware. The action determination device 10 may include a CPU, a GPU, a processor for learning processing, and a processor for inference processing as the processor 11. In this case, the skeleton extraction unit 22 that performs image processing may be implemented by a GPU; the learning unit 26 related to the learning of the learning model 32 may be implemented by a processor for learning processing; the action determination unit 24 that performs inference based on the learning model 32 may be implemented by a processor for inference processing; and the remaining image acquisition unit 21 and learning unit 26 may be implemented by a CPU.

Claims

1. An action determination device, comprising: An image acquisition unit acquires image data for each moment constituting the image data for learning in which an operator performing a series of operations including a plurality of operations is photographed, and acquires image data for each moment constituting the image data for learning and label information showing the content of the operator's actions at the respective moments; and acquires image data for each moment constituting image data in which an operator performing an unstable operation that is not normally performed in the series of operations is photographed, and label information showing the operator's actions shown in the image data at the respective moments when the unstable operation is performed; a skeleton extraction unit that extracts motion information from the image data at the target time, the motion information being skeleton information representing the posture of the operator and showing the relative positional relationship of a plurality of joints of the operator, taking the image data at each time point that constitutes the video data for learning acquired by the image acquisition unit as an object; and A learning unit, taking the image data at each moment as an object, makes a group of action information extracted by the skeleton extraction unit from the image data at the moment of the object and label information showing the action of the operator at the moment of the object as learning data, thereby generating a learning model that determines the action information similar to the input skeleton information when the skeleton information is input and outputs the label information corresponding to the determined action information, The image acquisition unit acquires image data of a subject, in which an operator performing a series of operations including a plurality of operations is photographed, at each time point constituting the image data of the subject. The skeleton extraction unit takes the image data at each moment constituting the image data of the object as an object, and extracts object information from the image data at the moment of the object, the object information being skeleton information representing the posture of the operator and showing the relative positional relationship of multiple joints of the operator, The action determination device further comprises: The action determination unit takes the image data at each moment constituting the image data of the object as an object, inputs the object information extracted from the image data at the moment of the object into the learning model generated by the learning unit, obtains the label information output from the learning model, determines the action content shown by the obtained label information as the action content being performed by the operator of the object, and when the operator of the object performs the unstable operation in a series of operations, determines that the operator of the object has performed the unstable operation.

2. A method for determining an action, wherein: With respect to image data for learning in which a worker performing a series of operations including a plurality of operations is photographed, an image acquisition unit of an action determination device acquires image data of each moment constituting the image data for learning and label information showing the content of the operator's actions at each moment; with respect to image data of each moment constituting image data in which a worker performing an unstable operation that is not normally performed in a series of operations is photographed, image data of the moment when the unstable operation is performed and label information showing the action of the worker shown in the image data at that moment are acquired, The skeleton extraction unit of the motion determination device takes the image data at each moment constituting the video data for learning as an object, and extracts motion information from the image data at the moment of the object, the motion information being skeleton information representing the posture of the operator and showing the relative positional relationship of multiple joints of the operator, The learning unit of the action determination device takes the image data of each moment as an object, so that a group of action information extracted by the skeleton extraction unit from the image data of the object at the moment and label information showing the action of the operator at the moment of the object is learned as learning data, thereby generating a learning model that determines the action information similar to the input skeleton information when the skeleton information is input and outputs the label information corresponding to the determined action information, The image acquisition unit of the action identification device acquires image data of each time point constituting the image data of the object, with respect to image data of an object in which an operator performing a series of operations including a plurality of operations is photographed. The skeleton extraction unit of the action determination device takes the image data at each moment constituting the image data of the object as an object, and extracts object information from the image data at the moment of the object, the object information being skeleton information representing the posture of the operator and showing the relative positional relationship of multiple joints of the operator, The action determination unit of the action determination device takes the image data at each moment constituting the image data of the object as the object, inputs the object information extracted from the image data at the moment of the object into the learning model, obtains the label information output from the learning model, and determines the action content shown by the obtained label information as the action content being performed by the operator of the object. When the operator of the object performs the unstable operation in a series of operations, it is determined that the operator of the object has performed the unstable operation.

3. A computer-readable storage medium storing an action determination program, the action determination program causing a computer to function as an action determination device that performs the following processing: Image acquisition processing, with respect to image data for learning in which an operator performing a series of operations including a plurality of operations is photographed, the image acquisition unit acquires image data at each moment constituting the image data for learning and label information showing the content of the operator's actions at each moment, and with respect to image data at each moment constituting image data in which an operator performing an unstable operation that is not normally performed in a series of operations is photographed, the image data at the moment when the unstable operation is performed and label information showing the operator's actions shown in the image data at that moment are acquired; a skeleton extraction process, wherein a skeleton extraction unit extracts motion information from the image data at each time point constituting the video data for learning acquired by the image acquisition process, the motion information being skeleton information representing the posture of the operator and showing the relative positional relationship of a plurality of joints of the operator; and The learning process takes the image data at each moment as an object, so that the action information extracted by the skeleton extraction unit from the image data at the moment of the object and the label information showing the action of the operator at the moment of the object are learned as learning data, thereby generating a learning model that determines the action information similar to the input skeleton information when the skeleton information is input and outputs the label information corresponding to the determined action information, In the image acquisition processing, with respect to image data of a subject whose operator performs a series of operations including a plurality of operations, image data at each time point constituting the image data of the subject are acquired. In the skeleton extraction process, the image data at each moment constituting the image data of the object is taken as an object, and object information is extracted from the image data at the moment of the object, the object information being skeleton information representing the posture of the operator and showing the relative positional relationship of multiple joints of the operator, Also conducted: An action determination process, wherein the action determination unit takes the image data at each moment constituting the image data of the object as an object, inputs the object information extracted from the image data at the moment of the object into the learning model generated by the learning process, obtains the label information output from the learning model, determines the action content shown by the obtained label information as the action content being performed by the operator of the object, and in the case where the operator of the object has performed the unstable operation in a series of operations, determines that the operator of the object has performed the unstable operation.

Images