Motion analysis device, program, and motion analysis method

Abstract

The motion analysis device (100) includes a joint position information acquisition unit (103) that acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, and uses one of one or more learned models for inferring the work being performed by the person from at least the positions of the joints of the person. A work identification unit (105) that infers the work being performed by the operator from the positions of the M joints and identifies the work being performed by the operator from the result of the inference, and when the confidence level of the inference is less than a predetermined threshold, N joints (N is an integer satisfying 1 ≦ N < M) are selected from the M joints, and a target selection unit (106) that uses the positions of the N joints as a target for use. The work identification unit (105) uses one of one or more learned models to infer again the work being performed by the operator from the target for use when the confidence level of the inference is less than a predetermined threshold.

Description

[Technical field]

[0001] The present disclosure relates to a motion analysis device, a program, and a motion analysis method. [Background technology]

[0002] At manufacturing production sites, task times are measured to improve manual work, etc. Task time measurement can be performed with little burden by automatically identifying the tasks performed by workers using image recognition. The posture analysis device described in Patent Document 1 identifies an activity from an activity image by using a trained posture model, area model, background model, and procedure model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-201772 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, even when performing the same task, differences in the joint position information indicating the positions of the worker's joints become large due to the worker's habits, resulting in inaccurate posture data of the worker. Therefore, if the worker in the image used for learning the model is different from the worker who identifies the task, the conventional technology may not be able to accurately identify the task procedure.

[0005] Therefore, one or more aspects of the present disclosure aim to enable accurate identification of tasks even if a worker has specific work habits. [Means for solving the problem]

[0006] The operation analysis device according to the first aspect of the present disclosure includes a joint position information acquisition unit that acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, and uses one of one or more learned models for inferring the operation being performed by the person from at least the positions of the joints of the person, infers the operation being performed by the operator from the positions of the M joints, and a work specifying unit that specifies the operation being performed by the operator from the result of the inference, and when the reliability of the inference is less than a predetermined threshold, selects N joints (N is an integer satisfying 1 ≦ N < M) from the M joints, and a use target selection unit that uses the positions of the N joints as use targets, and the work specifying unit, when the reliability of the inference is less than a predetermined threshold, uses one of the one or more learned models to infer again the operation being performed by the operator from the use target.

[0007] The operation analysis device according to the second aspect of the present disclosure includes a joint position information acquisition unit that acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, an object position information acquisition unit that acquires object position information indicating the position of an object included in the image, a distance calculation unit that calculates the distance between the position of the object and each of the positions of the M joints, a use target selection unit that selects N joints (N is an integer satisfying 1 ≦ N < M) from the joints where the distance is less than a predetermined threshold, and uses the positions of the N joints as use targets, and a work specifying unit that uses a learned model for inferring the operation being performed by the person from the positions of the joints of the person and the position of the object, infers the operation being performed by the operator from the use target, and specifies the operation being performed by the operator from the result of the inference.

[0008] The operation analysis device according to the third aspect of the present disclosure includes a joint position information acquisition unit that acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, an object position information acquisition unit that acquires object position information indicating the position of an object included in the image, a distance calculation unit that calculates the distance between the position of the object and each of the positions of the M joints, and when N joints (N is an integer satisfying 1 ≦ N < M) can be selected from the joints whose distance is less than a predetermined threshold value, the positions of the N joints selected from the joints whose distance is less than the predetermined threshold value are used as the objects of use, and when the N joints cannot be selected from the joints whose distance is less than the predetermined threshold value, the N joints are selected from predetermined joints, and a use target selection unit that uses the positions of the N joints as the use target, and uses one of one or more learned models for inferring the work performed by the person from at least the positions of the joints of the person, and infers the work performed by the operator from the use target, and a work specifying unit that specifies the work performed by the operator from the result of the inference.

[0009] The program according to the first aspect of the present disclosure causes a computer to function as a joint position information acquisition unit that acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, a work specifying unit that uses one of one or more learned models for inferring the work performed by the person from at least the positions of the joints of the person, infers the work performed by the operator from the positions of the M joints, and specifies the work performed by the operator from the result of the inference, and a use target selection unit that selects N joints (N is an integer satisfying 1 ≦ N < M) from the M joints and uses the positions of the N joints as the use target when the reliability of the inference is less than a predetermined threshold value, and when the reliability of the inference is less than a predetermined threshold value, the work specifying unit uses one of the one or more learned models to infer the work performed by the operator from the use target again.

[0010] The program according to the second aspect of the present disclosure causes a computer to function as a joint position information acquisition unit that acquires joint position information indicating the positions of M joints (where M is an integer of 2 or more) of an operator included in an image, an object position information acquisition unit that acquires object position information indicating the position of an object included in the image, a distance calculation unit that calculates the distance between the position of the object and each of the positions of the M joints, a usage target selection unit that selects N joints (where N is an integer satisfying 1 ≤ N < M) from the joints whose distance is less than a predetermined threshold value and uses the positions of the N joints as usage targets, and a work identification unit that infers the work being performed by the operator from the usage targets using a learned model for inferring the work being performed by the person from the positions of the joints of the person and the position of the object, and identifies the work being performed by the operator from the result of the inference.

[0011] The program according to the third aspect of the present disclosure causes a computer to function as a joint position information acquisition unit that acquires joint position information indicating the positions of M joints (where M is an integer of 2 or more) of an operator included in an image, an object position information acquisition unit that acquires object position information indicating the position of an object included in the image, a distance calculation unit that calculates the distance between the position of the object and each of the positions of the M joints, a usage target selection unit that, when it is possible to select N joints (where N is an integer satisfying 1 ≤ N < M) from the joints whose distance is less than a predetermined threshold value, uses the positions of the N joints selected from the joints whose distance is less than the predetermined threshold value as usage targets, and when it is not possible to select the N joints from the joints whose distance is less than the predetermined threshold value, selects the N joints from predetermined joints and uses the positions of the N joints as the usage targets, and a work identification unit that infers the work being performed by the operator from the usage targets using one of one or more learned models for inferring the work being performed by the person from at least the positions of the joints of the person, and identifies the work being performed by the operator from the result of the inference.

[0012] The operation analysis method according to the first aspect of the present disclosure acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, and uses one of one or more learned models for inferring the work being performed by the person from at least the positions of the joints of the person to infer the work being performed by the operator from the positions of the M joints, identify the work being performed by the operator from the result of the inference, and when the confidence level of the inference is less than a predetermined threshold, select N joints (N is an integer satisfying 1 ≦ N < M) from the M joints, use the positions of the N joints as the object of use, and when the confidence level of the inference is less than a predetermined threshold, use one of the one or more learned models to infer again the work being performed by the operator from the object of use.

[0013] The operation analysis method according to the second aspect of the present disclosure acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, acquires object position information indicating the position of an object included in the image, calculates the distance between the position of the object and each of the positions of the M joints, selects N joints (N is an integer satisfying 1 ≦ N < M) from the joints where the distance is less than a predetermined threshold, uses the positions of the N joints as the object of use, and uses a learned model for inferring the work being performed by the person from the positions of the joints of the person and the position of the object to infer the work being performed by the operator from the object of use, and identify the work being performed by the operator from the result of the inference.

[0014] The operation analysis method according to the third aspect of the present disclosure acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, acquires object position information indicating the position of an object included in the image, calculates the distance between the position of the object and each of the positions of the M joints, and when N joints (N is an integer satisfying 1 ≦ N < M) can be selected from the joints whose distance is less than a predetermined threshold, uses the positions of the N joints selected from the joints whose distance is less than the predetermined threshold as the objects to be used, and when the N joints cannot be selected from the joints whose distance is less than the predetermined threshold, selects the N joints from predetermined joints, uses the positions of the N joints as the objects to be used, and infers the operation being performed by the person from at least the positions of the joints of the person using one of one or more trained models, and specifies the operation being performed by the operator from the result of the inference.

Effect of the Invention

[0015] According to one or more aspects of the present disclosure, even if the operator has a habit in the operation, the operation can be accurately specified.

Brief Description of the Drawings

[0016] [Figure 1] It is a block diagram schematically showing the configuration of the operation analysis device according to Embodiment 1. [Diagram 2] It is a schematic diagram showing a first example of joint position information. [Diagram 3] It is a schematic diagram showing a second example of joint position information. [Figure 4] (A) and (B) are schematic diagrams showing a first example of the operator's movement during the learning of the operation identification model. [Diagram 5] (A) and (B) are schematic diagrams showing a second example of the operator's movement during the learning of the operation identification model. [Figure 6] It is a block diagram schematically showing the configuration of a PC. [Figure 7]4 is a flowchart showing the operation of the motion analysis device according to the first embodiment. [Figure 8] FIG. 11 is a block diagram illustrating a schematic configuration of a motion analysis device according to a second embodiment. [Figure 9] FIG. 11 is a schematic diagram showing an example of an operator's actions during inference. [Figure 10] 10 is a flowchart showing the operation of the motion analysis device according to the second embodiment. [Figure 11] FIG. 11 is a block diagram showing a schematic configuration of a motion analysis device according to a third embodiment. [Figure 12] FIG. 11 is a schematic diagram for explaining the distance between an object and the position of a joint. [Figure 13] 11 is a flowchart showing the operation of the motion analysis device according to the third embodiment. [Figure 14] FIG. 13 is a block diagram illustrating a schematic configuration of a motion analysis device according to a fourth embodiment. [Figure 15] 13 is a flowchart showing the operation of the motion analysis device according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Embodiment 1 FIG. 1 is a block diagram showing a schematic configuration of a motion analysis apparatus 100 according to the first embodiment. The motion analysis device 100 includes an image acquisition unit 101, a joint position identification model storage unit 102, a joint position information acquisition unit 103, a task identification model storage unit 104, a task identification unit 105, and a utilization target selection unit 106.

[0018] The image acquisition unit 101 acquires an image. For example, the image acquisition unit 101 may receive image data representing an image from a network via a communication unit (not shown), or may read out image data representing an image from a storage unit (not shown). The acquired image is provided to the joint position information acquisition unit 103.

[0019] The joint position identification model storage unit 102 stores a joint position identification model, which is a trained model for identifying the joint positions of a worker from an image. The joint position identification model is a model for inferring the joint positions of a worker included in an image from the image.

[0020] The joint position information acquisition unit 103 acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of the worker contained in the image from the image acquisition unit 101. For example, the joint position information acquiring unit 103 inputs the image from the image acquiring unit 101 into a joint position identifying model stored in the joint position identifying model storage unit 102, thereby identifying the joint position that is an inference result from the joint position identifying model. The joint whose position is to be identified here is determined in advance according to the work to be performed by the worker, etc. The inference result from the joint position identifying model is also assumed to include information for identifying the joint, such as the name of the joint, and the position of the joint.

[0021] Then, the joint position information acquisition unit 103 acquires the joint position information by generating joint position information indicating the identified joint position. Here, the joint position information may be, for example, information indicating the coordinates of the joints (here, the right wrist and the left wrist) at frame time f that identifies the task, as shown in FIG. 2, or may be, for example, a matrix of coordinates identified from images of frames included in a predetermined time N before and after frame time f that identifies the task, as shown in FIG. 3.

[0022] The task-specific model storage unit 104 stores one or more task-specific models, which are one or more trained models for inferring the task a person is performing from at least the positions of the person's joints. Here, as an example, it will be described assuming that one work identification model is stored. Note that a work identification model for identifying an operator's work from the positions of M joints (M is an integer of 2 or more), and a work identification model for identifying an operator's work from the positions of N joints (N is an integer satisfying 1 ≦ N < M) selected from the M joints may be stored in the work identification model storage unit 104.

[0023] The work identification unit 105 uses one of the one or more learned models stored in the work identification model storage unit 104 to infer the work being performed by the operator from the positions of the M joints, and identifies the work being performed by the operator from the result of the inference.

[0024] For example, when the work identification unit 105 acquires joint position information from the joint position information acquisition unit 103, first, by inputting the positions of the plurality of joints indicated by the joint position information into the work identification model stored in the work identification model storage unit 104, as the inference result, the work and the confidence level of the inference are acquired.

[0025] And when the confidence level as the inference result is equal to or higher than a predetermined threshold value, the work identification unit 105 identifies the work that is the inference result as the work of the operator. On the other hand, when the confidence level is less than the predetermined threshold value, the work identification unit 105 gives a request to select a target to be used to the target selection unit 106, for example, by turning on a target use flag. In this case, the work identification unit 105 gives the joint position information from the joint position information acquisition unit 103 to the target selection unit 106.

[0026] Next, as described below, when the task identification unit 105 receives a notification of a utilization target selected from the utilization target selection unit 106, it inputs the utilization target into the task identification model stored in the task identification model storage unit 104, thereby acquiring the task and the reliability of the inference as the inference result. Here, when the task identification unit 105 receives a notification of multiple utilization targets from the utilization target selection unit 106, it inputs each of the multiple utilization targets into the task identification model, thereby acquiring the task and the reliability of the inference as the inference result for each of the multiple utilization targets.

[0027] In other words, when the reliability of the inference is less than a predetermined threshold, the task identification unit 105 re-infers the task being performed by the worker from the target of use, using one of one or more trained models stored in the task identification model storage unit 104.

[0028] Then, when the reliability of the inference result for the target of use is equal to or greater than a predetermined threshold, the task identification unit 105 identifies the task that is the inference result for the target of use as the task of the worker. Here, multiple tasks may be identified. In addition, when the reliability of the inference result for the target of use is less than a predetermined threshold, the work identification unit 105 may identify one work selected from the works that are the inference result for the target of use as the worker's work, or may identify a predetermined number of works in order of increasing reliability. Furthermore, if the reliability of the inference result for the target of use is less than a predetermined threshold, the task identification unit 105 may not identify the task. In this case, the task identification unit 105 may identify the task being performed by the worker as an unknown task.

[0029] When the utilization target selection unit 106 receives a request from the task identification unit 105, it selects a utilization target that is an object to be used as an input to the trained model from the joint position information acquired by the joint position information acquisition unit 103. For example, when the reliability of the inference by the task identification unit 105 is less than a predetermined threshold, the utilization target selection unit 106 selects N joints from the M joints, and sets the positions of the N joints as utilization targets. Here, the utilization target selection unit 106 selects a predetermined joint from the multiple joints indicated by the joint position information, and selects the selected joint and its position as the utilization target. Then, the utilization target selection unit 106 notifies the task identification unit 105 of the selected utilization target.

[0030] The processing performed by the task specifying unit 105 and the utilization target selecting unit 106 will be described below using a concrete example. Here, it is assumed that the task-specific model is trained using the worker's wrist and its position as input data. When learning the task specific model, it is assumed that a worker 120 is assembling a product 130 by handling a first part 131 with his left hand while holding nothing in his right hand, as shown in FIG. 4(A).

[0031] In such a case, when inference is made using the task specific model, as shown in FIG. 4(B), if worker 121 handles a first part 131 with his left hand while simultaneously handling a second part 132 with his right hand when assembling product 130, the position of the wrist joint of the right hand of worker 120 at the time of learning will be significantly different from the position of the wrist joint of the right hand of worker 121 at the time of inference, and even if the wrist joint positions of the right hand and the wrist joint positions of the left hand are input into the task specific model, the reliability of inference made by the task specific model will be low.

[0032] Also, as shown in FIG. 5(A), when learning the task specific model, the worker 120 performs a task of handling the product 130 with his / her left hand, and when assembling the product 130, his / her free right hand is placed near the left hand.

[0033] In such a case, when inference is made using the task-specific model, as shown in FIG. 5(B), if worker 121 places his / her right hand away from his / her left hand while handling product 130 with his / her left hand, the position of the wrist joint of the right hand of worker 120 at the time of learning will be significantly different from the position of the wrist joint of the right hand of worker 121 at the time of inference, and even if the wrist joint positions of the right hand and the wrist joint positions of the left hand are input into the task-specific model, the reliability of inference by the task-specific model will be low.

[0034] Therefore, in the case shown in FIG. 4(B), the utilization object selection unit 106 selects each of the multiple joints, that is, the wrist joint of the right hand and the wrist joint of the left hand, as the utilization object, and the task identification unit 105 inputs only the position of the wrist joint of the right hand into the task identification model, and further inputs only the position of the wrist joint of the left hand into the task identification model, thereby making it possible to infer with a high degree of reliability the task of handling the first part 131 and the task of handling the second part 132.

[0035] Furthermore, in a case such as that shown in FIG. 5(B), the utilization object selection unit 106 selects each of the multiple joints, that is, the position of the wrist joint of the right hand and the position of the wrist joint of the left hand, as the utilization object, and the task identification unit 105 inputs only the position of the wrist joint of the right hand into the task identification model, and further inputs only the position of the wrist joint of the left hand into the task identification model, thereby making it possible to infer with a high degree of reliability the task of handling the first part 131. This is because it is possible to perform task inference using only the positions of the joints, which have a large influence on the inference of each task.

[0036] The above-described motion analysis apparatus 100 can be realized by a computer such as the PC 10 shown in FIG. The PC 10 includes storage 11 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), memory 12, a processor 13 such as a CPU (Central Processing Unit), a communication I / F (Interface) 14 such as a NIC (Network Interface Card), an input I / F 15 such as a keyboard and a mouse, and a display 16.

[0037] For example, the joint position identification model storage unit 102 and the task identification model storage unit 104 can be realized by the storage 11 or the memory 12. The image acquisition unit 101 , the joint position information acquisition unit 103 , the task identification unit 105 and the utilization target selection unit 106 can be realized by the processor 13 executing a program stored in the storage 11 .

[0038] The above programs may be downloaded to the storage 11 from a recording medium (not shown) via a reader / writer (not shown), or from a network via the communication I / F 14. The programs may also be directly loaded onto the memory 12 from a recording medium via the reader / writer, or from a network via the communication I / F 14, and executed by the processor 13. In other words, the program may be provided by a program product such as a recording medium.

[0039] FIG. 7 is a flowchart showing the operation of motion analysis apparatus 100 according to the first embodiment. First, the image acquisition unit 101 acquires an image (S10). The acquired image is provided to the joint position information acquisition unit 103.

[0040] The joint position information acquisition unit 103 acquires joint position information indicating the positions of a plurality of joints of the worker contained in the image from the image acquisition unit 101 (S11). For example, in the specific example shown in Fig. 4 or 5, the joint position information acquisition unit 103 acquires joint position information indicating the position of the wrist joint of the right hand and the position of the wrist joint of the left hand. Note that the joint position information also includes information for identifying the joints in association with their positions.

[0041] When task identification unit 105 acquires joint position information from joint position information acquisition unit 103, it first inputs the joint positions indicated in the joint position information into a task identification model stored in task identification model storage unit 104, and acquires the task and the reliability of the inference as an inference result (S12). Since the joint position information indicates the positions of multiple joints, the task is estimated by inputting the positions of the multiple joints into the task identification model.

[0042] The task identification unit 105 determines whether the reliability of the inference result of the inference in step S12 is equal to or greater than a predetermined threshold (S13). If the reliability is less than the predetermined threshold (NO in S13), the process proceeds to step S14, and if the reliability is equal to or greater than the predetermined threshold (YES in S13), the process proceeds to step S16.

[0043] In step S14, the utilization target selection unit 106, upon receiving a request from the task specification unit 105, selects a utilization target that is an object to be used as an input to the trained model from the joint position information acquired by the joint position information acquisition unit 103. Here, the utilization target selection unit 106 selects a predetermined joint that is a part of the multiple joints from the multiple joints indicated in the joint position information, and selects the selected joint and its position as the utilization target.

[0044] Then, when the task identification unit 105 receives the multiple use targets selected from the use target selection unit 106, it inputs each of the multiple use targets into the task identification model stored in the task identification model storage unit 104, and obtains the task and the reliability of the inference as the inference result (S15).Then, the process proceeds to step S16.

[0045] In step S16, if the reliability of the inference result for the target of use is equal to or greater than a predetermined threshold, the task identification unit 105 identifies the task that is the inference result for the target of use as the task of the worker. Here, multiple tasks may be identified. In addition, in step S15, if the reliability of the inference result for the target of use is less than a predetermined threshold, the work identification unit 105 may identify one work selected from the works that are the inference result for the target of use as the worker's work, or may identify a predetermined number of works in order of decreasing reliability. Furthermore, the task identification unit 105 may not identify a task if the reliability, which is an inference result for the target of use, is less than a predetermined threshold. In this case, the task identification unit 105 may treat the identified task or the inability to identify a task as a result of the operation, or may identify the task of the worker as an unknown task.

[0046] As described above, according to embodiment 1, even if the worker when making an inference performs a task using movements that are not seen in the worker during learning, the task can be identified with high accuracy by making an inference using some of the joints that have a large influence on the inference of the task.

[0047] Embodiment 2 FIG. 8 is a block diagram showing a schematic configuration of a motion analysis apparatus 200 according to the second embodiment. The motion analysis device 200 includes an image acquisition unit 101, a joint position identification model storage unit 102, a joint position information acquisition unit 103, a task identification model storage unit 204, a task identification unit 205, a usage target selection unit 206, an object position identification model storage unit 207, and an object position information acquisition unit 208.

[0048] The image acquisition unit 101, the joint position identification model storage unit 102, and the joint position information acquisition unit 103 of the motion analysis apparatus 200 of embodiment 2 are similar to the image acquisition unit 101, the joint position identification model storage unit 102, and the joint position information acquisition unit 103 of the motion analysis apparatus 100 of embodiment 1. However, the image acquiring unit 101 in the second embodiment also provides the acquired image to the object position information acquiring unit 208 .

[0049] The object position identification model storage unit 207 stores an object position identification model, which is a trained model for identifying an object position, which is the position of an object, from an image. The object position identification model is a model for inferring the position of an object included in an image from the image.

[0050] The object position information acquisition unit 208 acquires object position information indicating the position of an object included in the image from the image acquisition unit 101 . For example, the object position information acquisition unit 208 inputs the image from the image acquisition unit 101 into the object position identification model stored in the object position identification model storage unit 207, thereby identifying the object position, which is an inference result from the object position identification model. The object whose position is to be identified here is determined in advance according to the work performed by the worker. The inference result from the object position identification model is also assumed to include information for identifying the object and the position of the object.

[0051] Then, the object position information acquisition unit 208 acquires the object position information by generating object position information indicating the identified object position. Here, the object position information may be, for example, information indicating the coordinates of an object at frame time f at which the object is identified, as in the description in Figure 2, or may be, for example, a matrix of coordinates identified from images of frames included in a predetermined time N before or after frame time f at which the object is identified, as in the description in Figure 3.

[0052] The task specific model storage unit 204 receives the joint positions of the worker as input and stores a plurality of task specific models that specify the tasks of the worker. In the second embodiment, the task specific model storage unit 204 stores a multi-joint task specific model for inferring a worker's task from the positions of multiple joints, and a combination task specific model for inferring a worker's task for each predetermined combination selected from a plurality of objects and a plurality of joints.

[0053] When the task identification unit 205 acquires joint position information from the joint position information acquisition unit 103, it first inputs the positions of the multiple joints indicated in the joint position information into a multiple-joint task identification model stored in the task identification model storage unit 204, and acquires, as an inference result, a task and the reliability of the inference.

[0054] Then, when the reliability of the inference result is equal to or greater than a predetermined threshold, the task identification unit 205 identifies the task that is the inference result as the task of the worker. On the other hand, if the reliability is less than a predetermined threshold, the task identification unit 205, for example, turns on a usage target flag to provide a request to select a usage target to the usage target selection unit 206. In this case, the task identification unit 205 provides the joint position information from the joint position information acquisition unit 103 to the usage target selection unit 206.

[0055] Next, when the task identification unit 205 receives the multiple utilization targets selected from the utilization target selection unit 206, it inputs each of the multiple utilization targets into the corresponding combination task identification model stored in the task identification model storage unit 204, and obtains the task and the reliability of the inference as the inference result. Then, when the reliability of the inference result for the target of use is equal to or greater than a predetermined threshold, the task identification unit 205 identifies the task, which is the inference result for the target of use, as the task of the worker. Here, multiple tasks may be identified. In addition, when the reliability of the inference result for the target of use is less than a predetermined threshold, the work identification unit 205 may identify one work selected from the works that are the inference result for the target of use as the worker's work, or may identify a predetermined number of works in order of increasing reliability. Furthermore, if the reliability of the inference result for the target of use is less than a predetermined threshold, the task identification unit 205 may not identify the task. In this case, the task identification unit 205 may identify the task of the worker as an unknown task.

[0056] When the utilization target selection unit 206 receives a request from the task specification unit 205, the utilization target selection unit 206 selects a utilization target that is an object to be used as an input to the trained model from the joint position information acquired by the joint position information acquisition unit 103 and the object position information acquired by the object position information acquisition unit 208. For example, the utilization target selection unit 206 selects N joints from the joints previously associated with the object indicated by the object position information, and sets the positions of the N joints and the object positions as utilization targets. Here, the utilization target selection unit 206 may select a predetermined combination from the multiple joints indicated by the joint position information and the multiple objects indicated by the object position information, and select the selected combination and the positions of the joints and objects included in the combination as utilization targets. Then, the utilization target selection unit 206 notifies the task identification unit 205 of the selected utilization target.

[0057] The processing performed by the task identifying unit 205 and the utilization target selecting unit 206 will be described below using a concrete example. Here, we will explain the case where a multi-joint task specific model is trained using the position of the worker's wrist joint as input data, and a combination task specific model is trained using the combination of the position of the worker's wrist joint and the position of a part as an object as input data. Again, when learning the multi-joint task specific model, it is assumed that the worker 120 is assembling the product 130 by handling the first part 131 with the left hand while holding nothing in the right hand, as shown in FIG. 4(A).

[0058] In such a case, when inference is made using the task specific model, as shown in FIG. 9, if worker 122 handles a first part 131 with his left hand while simultaneously handling a second part 132 with his right hand when assembling product 130, the position of the wrist joint of the right hand of worker 120 at the time of learning will be significantly different from the position of the wrist joint of the right hand of worker 122 at the time of inference, and even if the positions of the wrist joint of the right hand and the wrist joint of the left hand are input into the task specific model, the reliability of inference made by the task specific model will be low.

[0059] Therefore, in the second embodiment, the use target selection unit 206 selects, as the use targets, a combination of the position of the left hand joint and the position of the first part 131, and a combination of the position of the right hand joint and the position of the second part 132.

[0060] Then, the task identification unit 205 uses a combination task identification model, which is a trained model trained using the combination of the positions of the joints of the left hand and the position of the object corresponding to the first part 131 as input data, to perform inference using the combination of the positions of the joints of the left hand and the position of the first part 131 as input, thereby making it possible to infer the task of handling the first part 131 with a high degree of reliability.

[0061] In addition, the task identification unit 205 uses a combination task identification model, which is a trained model trained using the combination of the positions of the joints of the right hand and the position of the object corresponding to the second part 132 as input data, to perform inference using the combination of the positions of the joints of the right hand and the position of the second part 132 as input, thereby making it possible to infer the task of handling the second part 132 with a high degree of reliability.

[0062] The above-described motion analysis apparatus 200 can also be realized by a computer such as the PC 10 shown in FIG. For example, the object localization model storage unit 207 can be realized by the storage 11 or the memory 12. The object position information acquisition unit 208 can be realized by the processor 13 executing a program stored in the storage 11.

[0063] FIG. 10 is a flowchart showing the operation of motion analysis apparatus 200 according to the second embodiment. In FIG. 10, steps that perform the same processes as those in the flowchart shown in FIG. 7 are assigned the same reference numerals as those in the flowchart shown in FIG. 7, and detailed descriptions thereof will be omitted.

[0064] The processing in steps S10 to S13 and step S16 in the flowchart shown in FIG. 10 are similar to the processing in steps S10 to S13 and step S16 in the flowchart shown in FIG. However, in FIG. 10, if the reliability is less than a predetermined threshold value (NO in S13), the process proceeds to step S24. In addition, in FIG. 10, the image acquired in step S10 is provided to the joint position information acquisition unit 103 and the object position information acquisition unit 208.

[0065] In step S24, the object position information acquisition unit 208 acquires object position information indicating the positions of objects included in the image from the image acquisition unit 101. For example, in the specific example shown in FIG. 9, the object position information acquisition unit 208 acquires object position information indicating the positions of the first part 131 and the second part 132.

[0066] Next, upon receiving a request from the task specifying unit 205, the utilization target selecting unit 206 selects a utilization target to be used as an input to the trained model from the joint position information acquired by the joint position information acquiring unit 103 and the object position information acquired by the object position information acquiring unit 208. In the second embodiment, the utilization target selecting unit 206 selects a predetermined combination from a plurality of joints indicated in the joint position information and at least one article indicated in the object position information, and selects the joints, the article, and their positions of the selected combination as utilization targets.

[0067] Then, when the task identification unit 205 receives the selected use object from the use object selection unit 206, it inputs the use object into the corresponding combination task identification model stored in the task identification model storage unit 204, and obtains the task and the reliability of the inference as the inference result (S26).Then, the process proceeds to step S16.

[0068] As described above, according to the second embodiment, even if the worker when making an inference performs a task using movements not seen by the worker during learning, the task can be identified with high accuracy by making an inference using some of the joints and objects that have a large influence on the inference of the task.

[0069] Embodiment 3 FIG. 11 is a block diagram showing a schematic configuration of a motion analysis apparatus 300 according to the third embodiment. The motion analysis device 300 includes an image acquisition unit 101, a joint position identification model storage unit 102, a joint position information acquisition unit 103, a task identification model storage unit 304, a task identification unit 305, a usage target selection unit 306, an object position identification model storage unit 207, an object position information acquisition unit 208, and a distance calculation unit 309.

[0070] The image acquisition unit 101, the joint position identification model storage unit 102, and the joint position information acquisition unit 103 of the motion analysis apparatus 300 according to embodiment 3 are similar to the image acquisition unit 101, the joint position identification model storage unit 102, and the joint position information acquisition unit 103 of the motion analysis apparatus 100 according to embodiment 1. However, the image acquiring unit 101 in the third embodiment also provides the acquired image to the object position information acquiring unit 208. Moreover, the joint position information acquiring unit 103 in the third embodiment provides the acquired joint position information to the distance calculation unit 309.

[0071] The object position identification model storage unit 207 and the object position information acquisition unit 208 of the motion analysis apparatus 300 according to the third embodiment are similar to the object position identification model storage unit 207 and the object position information acquisition unit 208 of the motion analysis apparatus 200 according to the second embodiment. However, the object position information acquisition unit 208 in the third embodiment provides the acquired object position information to the distance calculation unit 309 .

[0072] The distance calculation unit 309 calculates the distance between the object position and each of the M joint positions. For example, the distance calculation unit 309 calculates the distance for each combination of the positions of multiple joints indicated in the joint position information and the position of at least one object indicated in the object position information. The calculated distances are notified to the target selection unit 306.

[0073] The utilization target selection unit 306 selects N joints from among the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, and sets the positions of the N joints and the position of the object indicated by the object position information as utilization targets. For example, the utilization target selection unit 306 refers to the distance from the distance calculation unit 309, identifies a joint whose distance from the object is closer than a predetermined threshold, and selects the object, the identified joint, and their positions as utilization targets. Then, the utilization target selection unit 306 notifies the task identification unit 205 of the selected utilization target.

[0074] The task-specific model storage unit 304 stores a task-specific model, which is a trained model for inferring the task a person is performing from the positions of the person's joints and the positions of an object.

[0075] The task identification unit 305 uses the trained model stored in the task identification model storage unit 304 to infer the task being performed by the worker from the use object notified by the use object selection unit 306, and identifies the task being performed by the worker from the result of the inference. Here, when the task identification unit 305 receives notification of the use object selected from the use object selection unit 306, it inputs the use object into the task identification model stored in the task identification model storage unit 304, thereby acquiring the task and the reliability of the inference as the inference result. Then, the task identification unit 305 identifies the acquired task as the task being performed by the worker.

[0076] The processes performed by the distance calculation unit 309, the task identification unit 305, and the utilization target selection unit 306 will be described below using specific examples. Here, it is assumed that a task-specific model is trained using a combination of the position of the worker's wrist joint and the position of an object as input data. In such a case, when making an inference using the task-specific model, if the position of the right hand, which is not involved in the task, is different from that during learning, even if the positions of the wrist joints of the right hand and the wrist joints of the left hand are input into the task-specific model, the reliability of the inference made by the task-specific model may be low.

[0077] In the third embodiment, as shown in FIG. 12, a distance D1 between the product 130 as an object and the position of the joint of the left hand, and a distance D2 between the product 130 and the position of the joint of the right hand are calculated by the distance calculation unit 309.

[0078] Then, the usage target selection unit 306 compares the distance D1 and the distance D2 calculated by the distance calculation unit 309 with a predetermined threshold value, and selects as the usage target the position of the product 130 and the position of the left hand joint corresponding to the distance D1 that is less than the threshold value.

[0079] Then, the task identification unit 305 uses a combination task identification model, which is a trained model trained using a combination of the positions of the joints of the left hand and the position of an object as input data, to perform inference using the combination of the positions of the joints of the left hand and the position of the product 130 as input, thereby making it possible to infer the task of handling the product 130 with a high degree of reliability.

[0080] The above-described motion analysis apparatus 300 can also be realized by a computer such as the PC 10 shown in FIG. For example, the distance calculation unit 309 can also be realized by the processor 13 executing a program stored in the storage 11.

[0081] FIG. 13 is a flowchart showing the operation of motion analysis apparatus 300 according to the third embodiment. In FIG. 13, steps that perform the same processes as those in the flowchart shown in FIG. 7 are assigned the same reference numerals as those in the flowchart shown in FIG. 7, and detailed descriptions thereof will be omitted.

[0082] The processes in steps S10 and S11 in the flowchart shown in FIG. 13 are similar to the processes in steps S10 and S11 in the flowchart shown in FIG. 13, after the process in step S11, the process proceeds to step S32. The joint position information acquired in step S11 is provided to the distance calculation unit 309.

[0083] In step S32, the object position information acquisition unit 208 acquires object position information indicating the position of an object included in the image from the image acquisition unit 101. For example, in the specific example shown in Fig. 12, the object position information acquisition unit 208 acquires object position information indicating the position of the product 130. The acquired object position information is provided to the distance calculation unit 309.

[0084] The distance calculation unit 309 calculates the distance for each combination of the positions of the multiple joints indicated by the joint position information and the position of at least one object indicated by the object position information (S33). The calculated distances are notified to the target selection unit 306.

[0085] The target to be used selection unit 306 refers to the distance notified by the distance calculation unit 309, identifies a joint whose distance from the object is closer than a predetermined threshold, and selects the object, the identified joint, and their positions as targets to be used (S34). Then, the utilization target selection unit 306 notifies the task identification unit 205 of the selected utilization target.

[0086] When the task identification unit 305 receives notification of the selected target of use from the target of use selection unit 306, it inputs the target of use into the task identification model stored in the task identification model storage unit 304, and obtains the task and the reliability of the inference as the inference result (S35).

[0087] Then, the task identification unit 105 identifies the task that is the inference result for the target of use as the task of the worker (S36). Here, a plurality of tasks may be identified.

[0088] As described above, according to the third embodiment, even if the worker when making an inference performs a task using movements not seen by the worker during learning, the task can be identified with high accuracy by making an inference using some of the joints and objects that have a large influence on the inference of the task.

[0089] 13, the utilization target selection unit 306 refers to the distance from the distance calculation unit 309 and selects, as the utilization target, a joint whose distance from the object is closer than a predetermined threshold value, but the third embodiment is not limited to this example. For example, the utilization target selection unit 306 may refer to the distance from the distance calculation unit 309 and select, as the utilization target, a joint whose distance from the object is closest.

[0090] In the third embodiment described above, the target to be used selection unit 306 selects N joints from among the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, and sets the positions of the N joints and the position of the object indicated by the object position information as targets to be used, but the third embodiment is not limited to this example. For example, the utilization target selection unit 306 may select N joints from among the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, and set the positions of the N joints as utilization targets.

[0091] In the third embodiment, a use object is selected based on the distance calculated by the distance calculation unit 309, and a task is identified based on the selected use object, but the third embodiment is not limited to this example. For example, similarly to the first embodiment, the task identification unit 305 may infer a task based on the positions of multiple joints, and if the reliability of the inferred task is less than a predetermined threshold, the processing of steps S32 to S36 shown in FIG. 13 may be performed. In other words, when the reliability of the inference by the task specifying unit 305 is less than a predetermined threshold, the utilization target selecting unit 306 in the third embodiment selects N joints from among the joints whose distances calculated by the distance calculating unit 309 are less than a predetermined threshold, and sets the positions of the N joints and the position of the object as utilization targets. The N joints selected here are assumed to be predetermined. Then, the task specifying unit 105 performs inference using the utilization targets. In this case, the utilization target selecting unit 306 does not need to set the position of the object as the utilization target.

[0092] Embodiment 4 FIG. 14 is a block diagram showing a schematic configuration of a motion analysis apparatus 400 according to the fourth embodiment. The motion analysis device 400 includes an image acquisition unit 101, a joint position identification model storage unit 102, a joint position information acquisition unit 103, a task identification model storage unit 404, a task identification unit 405, a usage target selection unit 406, an object position identification model storage unit 207, an object position information acquisition unit 208, a distance calculation unit 309, a correspondence table storage unit 410, and a task limitation unit 411.

[0093] The image acquisition unit 101, the joint position identification model storage unit 102, and the joint position information acquisition unit 103 of the motion analysis apparatus 400 of embodiment 4 are similar to the image acquisition unit 101, the joint position identification model storage unit 102, and the joint position information acquisition unit 103 of the motion analysis apparatus 100 of embodiment 1. However, the image acquiring unit 101 in the fourth embodiment also provides the acquired image to the object position information acquiring unit 208. Moreover, the joint position information acquiring unit 103 in the fourth embodiment provides the acquired joint position information to the distance calculation unit 309.

[0094] The object position identification model storage unit 207 and the object position information acquisition unit 208 of the motion analysis apparatus 400 according to the fourth embodiment are similar to the object position identification model storage unit 207 and the object position information acquisition unit 208 of the motion analysis apparatus 200 according to the second embodiment. However, the object position information acquisition unit 208 in the fourth embodiment provides the acquired object position information to the distance calculation unit 309 and the operation limitation unit 411 .

[0095] The task specific model storage unit 304 and the distance calculation unit 309 of the motion analysis apparatus 400 according to the fourth embodiment are similar to the task specific model storage unit 304 and the distance calculation unit 309 of the motion analysis apparatus 300 according to the third embodiment.

[0096] When N joints can be selected from the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, the target selection unit 406 selects the positions of the N joints selected from the joints whose distances are less than the predetermined threshold and the position of the object indicated by the object position information as targets for use. The N joints selected here are assumed to be predetermined according to an object that is at a distance less than the threshold. On the other hand, if the utilization target selection unit 406 cannot select N joints from the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, it selects N joints from predetermined joints and sets the positions of the N joints as utilization targets.

[0097] For example, the target to be used selection unit 406 refers to the distance from the distance calculation unit 309, identifies N joints whose distance from the object is closer than a predetermined threshold, and selects the object, the identified joints, and their positions as targets to be used. Then, the utilization target selection unit 406 notifies the task identification unit 405 and the task limitation unit 411 of the selected utilization target.

[0098] In addition, the target of use selection unit 406 refers to the distance from the distance calculation unit 309, and if there are no N joints whose distance from the object is closer than a predetermined threshold, selects a target of use that is to be used as an input to the trained model from the joint position information acquired by the joint position information acquisition unit 103, as in embodiment 1. Then, the utilization target selection unit 406 notifies the task identification unit 405 of the selected utilization target.

[0099] The correspondence table storage unit 410 is a correspondence information storage unit that stores a correspondence table, which is correspondence information that associates each of a plurality of tasks with at least one of a plurality of objects. In the correspondence table, tasks are associated with objects related to the tasks.

[0100] When N joints can be selected from the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, the task limiting unit 411 refers to the correspondence information stored in the correspondence table storage unit 410 to identify one or more tasks associated with the object indicated by the object position information as candidate tasks. For example, the task limiting unit 411 refers to the correspondence table stored in the correspondence table storage unit 410 to identify tasks associated with objects included in the use target notified by the use target selection unit 406, and notifies the task identification unit 405 of the identified tasks as candidate tasks.

[0101] The task specific model storage unit 404 receives the joint position information of the worker from the joint position information and stores one or more task specific models that specify the task of the worker. The one or more task specific models here include a joint task specific model, which is a trained model similar to that in the first embodiment, and a combination task specific model, which is a trained model for inferring the task of the worker from a combination of the position of an object and the position of a joint.

[0102] When the task identification unit 405 receives notification of the selected target of use from the target of use selection unit 406, it inputs the target of use into a joint task identification model or a combination task identification model stored in the task identification model storage unit 404 according to the target of use, and obtains the task and the reliability of the inference as the inference result. When the combination task identification model is used, if the task as the inference result matches a candidate task notified by the task limitation unit 411, the task identification unit 405 identifies the task matching the candidate task as the worker's task. In other words, when N joints can be selected from the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, the task identification unit 405 identifies a task that is inferred from the target of use and matches a candidate task as the worker's task.

[0103] When the joint task identification model is used, the task identification unit 405 identifies the task as the inference result as the task of the worker.

[0104] The above-described motion analysis apparatus 400 can also be realized by a computer such as the PC 10 shown in FIG. For example, the correspondence table storage unit 410 can also be realized by the storage 11 or the memory 12. The operation limiting unit 411 can also be realized by the processor 13 executing a program stored in the storage 11.

[0105] FIG. 15 is a flowchart showing the operation of motion analysis apparatus 400 according to the fourth embodiment. In FIG. 15, steps that perform the same processes as those in the flowchart shown in FIG. 7 are assigned the same reference numerals as those in the flowchart shown in FIG. 7, and detailed descriptions thereof will be omitted.

[0106] The processes in steps S10 and S11 in the flowchart shown in FIG. 15 are similar to the processes in steps S10 and S11 in the flowchart shown in FIG. 13, after the process in step S11, the process proceeds to step S42. The joint position information acquired in step S11 is provided to the distance calculation unit 309.

[0107] In step S42, the object position information acquisition unit 208 acquires object position information indicating the position of an object included in the image from the image acquisition unit 101. The acquired object position information is provided to the distance calculation unit 309.

[0108] The distance calculation unit 309 calculates the respective distances in every combination of the positions of a plurality of joints indicated by the joint position information and the position of at least one object indicated by the object position information (S43). The calculated distances are notified to the usage target selection unit 406.

[0109] The usage target selection unit 406 refers to the distances notified from the distance calculation unit 309 and determines whether there are N predetermined joints whose distances from the object are closer than a predetermined threshold (S44). If there are N predetermined joints whose distances from the object are closer than the predetermined threshold (YES in S44), the process proceeds to step S45. If there are no N predetermined joints whose distances from the object are closer than the predetermined threshold (NO in S44), the process proceeds to step S48.

[0110] In step S45, the usage target selection unit 406 refers to the distances from the distance calculation unit 309, specifies N predetermined joints whose distances from the object are closer than the predetermined threshold, and selects the object, the specified joints, and their positions as the usage targets. Then, the usage target selection unit 406 notifies the selected usage targets to the work specification unit 405 and the work limitation unit 411.

[0111] When the work limitation unit 411 receives the notification of the usage target from the usage target selection unit 406, it refers to the correspondence table stored in the correspondence table storage unit 410 to specify the work associated with the object included in the usage target notified from the usage target selection unit 406, and notifies the specified work to the work specification unit 405 as the candidate work (S46).

[0112] When the work specification unit 405 receives the notification of the usage target selected from the usage target selection unit 306, it inputs the usage target into the combined work specification model stored in the work specification model storage unit 404, and as the inference result, obtains the work and the reliability of the inference (S47). Then, the process proceeds to step S50.

[0113] On the other hand, in step S44, if there are no predetermined N joints whose distance from the object is closer than the predetermined threshold (NO in S44), the process proceeds to step S48, and the utilization target selection unit 406 selects a utilization target that is an object to be used as an input to the trained model, from the joint position information acquired by the joint position information acquisition unit 103. Here, similar to the first embodiment, the utilization target selection unit 406 selects a predetermined joint that is a part of the multiple joints from the multiple joints indicated by the joint position information, and selects the selected joint and its position as the utilization target.

[0114] Then, when the task identification unit 405 receives the multiple use objects selected from the use object selection unit 406, it inputs each of the multiple use objects into the joint task identification model stored in the task identification model storage unit 404, and obtains the task and the reliability of the inference as the inference result (S46).Then, the processing proceeds to step S50.

[0115] In step S50, the task identification unit 405 identifies the task inferred as the task of the worker. Here, a plurality of tasks may be identified.

[0116] As described above, according to the fourth embodiment, even if the worker when making an inference performs a task using movements not seen by the worker during learning, the task can be identified with high accuracy by making an inference using some of the joints and objects that have a large influence on the inference of the task. Furthermore, in the fourth embodiment, since the candidate tasks are limited depending on the items used in the tasks, the tasks can be identified with high accuracy.

[0117] Here, in the fourth embodiment, when N joints can be selected from the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, the target to be used selection unit 406 selects the positions of the N joints selected from the joints whose distances are less than the predetermined threshold and the position of the object indicated by the object position information as the targets to be used; however, the fourth embodiment is not limited to such an example. For example, if the target to be used selection unit 406 can select N joints from among the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, the target to be used may be the positions of the N joints selected from among the joints whose distances are less than the predetermined threshold.

[0118] In the fourth embodiment, the process of selecting a utilization target according to the distance calculated by the distance calculation unit 309 is performed, but the fourth embodiment is not limited to this example. For example, similarly to the first embodiment, the task identification unit 405 may infer a task based on the positions of multiple joints, and if the reliability of the inferred task is less than a predetermined threshold, the processing of steps S42 to S50 shown in FIG. 15 may be performed.

[0119] In other words, when N joints can be selected from the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, the utilization target selection unit 406 in the fourth embodiment sets the positions of the N joints selected from the joints whose distances are less than the predetermined threshold and the position of the object as utilization targets. Note that even in such a case, the utilization target selection unit 406 does not need to include the position of the object in the utilization targets. On the other hand, when the utilization target selection unit 406 cannot select N joints from the joints whose distances are less than the predetermined threshold, it selects predetermined joints as the N joints and sets the positions of the N joints as utilization targets. Note that, when the joints whose distances calculated by the distance calculation unit 309 are less than the predetermined threshold include the predetermined N joints, the utilization target selection unit 406 may determine that it is possible to select N joints from the joints whose distances calculated by the distance calculation unit 309 are less than the predetermined threshold.

[0120] Then, when N joints can be selected from the joints whose distances calculated by the distance calculation unit 309 are less than a predetermined threshold, the task limiting unit 411 refers to correspondence information that associates each of the multiple tasks with at least one of the multiple objects, thereby identifying one or more tasks associated with the object indicated by the object position information as candidate tasks. When the task inferred from the target of use matches such a candidate task, the task identifying unit 405 identifies the task matching the candidate task as the task of the worker.

[0121] The joint position information acquiring unit 103 in the above-described embodiments 1 to 4 identifies a joint position using a joint position identifying model stored in the joint position identifying model storage unit 102, but the embodiments 1 to 4 are not limited to such examples. For example, the joint position information acquiring unit 103 may identify a joint position by using a conventional technique such as pattern matching, and acquire joint position information indicating the identified joint position.

[0122] The object position information acquiring unit 208 in the above-described embodiments 2 to 4 identifies the object position using the object position identification model stored in the object position identification model storage unit 207, but the embodiments 2 to 4 are not limited to such examples. For example, the object position information acquiring unit 208 may identify the object position by using a conventional technique such as pattern matching, and acquire object position information indicating the identified object position. [Explanation of symbols]

[0123] 100,200,300,400 Motion analysis device, 101 Image acquisition unit, 102 Joint position identification model memory unit, 103 Joint position information acquisition unit, 104,204,304,404 Task identification model memory unit, 105,205,305,405 Task identification unit, 106,206,306,406 Usage target selection unit, 207 Object position identification model memory unit, 208 Object position information acquisition unit, 309 Distance calculation unit, 410 Correspondence table memory unit, 411 Task limitation unit.

Claims

1. a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of a worker included in an image; an operation identification unit that infers an operation performed by the worker from the positions of the M joints using one of one or a plurality of trained models for inferring an operation performed by the worker from the positions of at least the joints of the worker, and identifies the operation performed by the worker from a result of the inference; a utilization target selection unit that selects N joints (N is an integer satisfying 1≦N<M) from the M joints when the reliability of the inference is less than a predetermined threshold, and sets positions of the N joints as utilization targets; When the reliability of the inference is less than a predetermined threshold, the task identification unit re-infers the task being performed by the worker from the target of use by using one of the one or more trained models. A motion analysis device comprising:

2. The utilization target selection unit selects predetermined joints as the N joints. The motion analysis device according to claim 1 .

3. An object position information acquisition unit that acquires object position information indicating a position of an object included in the image, The utilization target selection unit selects joints that are previously associated with the object as the N joints, and sets positions of the N joints and a position of the object as the utilization targets. The motion analysis device according to claim 1 .

4. an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculation unit that calculates a distance between a position of the object and each of the M joint positions, The utilization target selection unit selects, as the N joints, joints whose distances are less than a predetermined threshold value. The motion analysis device according to claim 1 .

5. The utilization target selection unit also selects the position of the object as the utilization target. The motion analysis device according to claim 4 .

6. an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculation unit that calculates a distance between a position of the object and each of the M joint positions, When the N joints can be selected from the joints whose distance is less than a predetermined threshold, the utilization target selection unit sets the positions of the N joints selected from the joints whose distance is less than a predetermined threshold as the utilization targets, and when the N joints cannot be selected from the joints whose distance is less than a predetermined threshold, the utilization target selection unit selects predetermined joints as the N joints and sets the positions of the N joints as the utilization targets. The motion analysis device according to claim 1 .

7. The utilization target selection unit selects the N joints from among the joints whose distance is less than a predetermined threshold value, and the position of the object is also selected as the utilization target. The motion analysis device according to claim 6 .

8. a task limiting unit that, when the N joints can be selected from the joints whose distance is less than a predetermined threshold, identifies one or more tasks associated with the object as candidate tasks by referring to correspondence information that associates each of a plurality of tasks with at least one of a plurality of objects; The task identification unit, when the N joints can be selected from the joints whose distance is less than a predetermined threshold, identifies a task that is inferred from the target of use and matches the candidate task as the task of the worker. The motion analysis device according to claim 6 or 7,

9. a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of a worker included in an image; an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculation unit that calculates a distance between a position of the object and each of the M joint positions; a utilization target selection unit that selects N joints (N is an integer satisfying 1≦N<M) from among the joints whose distance is less than a predetermined threshold, and sets the positions of the N joints as utilization targets; and a task identification unit that uses a trained model for inferring a task being performed by a person from the positions of the person's joints and the positions of an object, infers a task being performed by the person from the target of use, and identifies the task being performed by the person from the result of the inference. A motion analysis device comprising:

10. a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of a worker included in an image; an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculation unit that calculates a distance between a position of the object and each of the M joint positions; a utilization object selection unit that, when N joints (N is an integer satisfying 1≦N<M) can be selected from the joints whose distance is less than a predetermined threshold, sets the positions of the N joints selected from the joints whose distance is less than a predetermined threshold to the utilization object, and, when the N joints cannot be selected from the joints whose distance is less than a predetermined threshold, selects the N joints from predetermined joints and sets the positions of the N joints to the utilization object; and a task identification unit that infers the task being performed by the worker from the target of use by using one of one or a plurality of trained models for inferring the task being performed by the worker from at least the positions of the joints of the worker, and identifies the task being performed by the worker from the result of the inference. A motion analysis device comprising:

11. a task limiting unit that, when the N joints can be selected from the joints whose distance is less than a predetermined threshold, identifies one or more tasks associated with the object as candidate tasks by referring to correspondence information that associates each of a plurality of tasks with at least one of a plurality of objects; The task identification unit, when the N joints can be selected from the joints whose distance is less than a predetermined threshold, identifies a task that is inferred from the target of use and matches the candidate task as the task of the worker. The motion analysis device according to claim 10 .

12. Computer, a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of the worker included in the image; an operation identification unit that uses one of one or more trained models for inferring an operation performed by a person from positions of the person's joints, infers an operation performed by the person from the positions of the M joints, and identifies the operation performed by the person from a result of the inference; and when the reliability of the inference is less than a predetermined threshold, the unit functions as a utilization target selection unit that selects N joints (N is an integer satisfying 1≦N<M) from the M joints and sets the positions of the N joints as utilization targets; When the reliability of the inference is less than a predetermined threshold, the task identification unit re-infers the task being performed by the worker from the target of use by using one of the one or more trained models. A program characterized by.

13. Computer, a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of the worker included in the image; an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculation unit that calculates a distance between a position of the object and each of the M joint positions; a utilization target selection unit that selects N joints (N is an integer satisfying 1≦N<M) from among the joints whose distance is less than a predetermined threshold, and sets the positions of the N joints as utilization targets; and and functioning as a task identification unit that infers the task being performed by the worker from the target of use using a trained model for inferring the task being performed by the person from the positions of the person's joints and the positions of an object, and identifies the task being performed by the worker from the result of the inference. A program characterized by.

14. Computer, a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of the worker included in the image; an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculation unit that calculates a distance between a position of the object and each of the M joint positions; a utilization object selection unit that, when N joints (N is an integer satisfying 1≦N<M) can be selected from the joints whose distance is less than a predetermined threshold, sets the positions of the N joints selected from the joints whose distance is less than a predetermined threshold to the utilization object, and, when the N joints cannot be selected from the joints whose distance is less than a predetermined threshold, selects the N joints from predetermined joints and sets the positions of the N joints to the utilization object; and and functioning as a task identification unit that infers the task being performed by the worker from the target of use by using one of one or a plurality of trained models for inferring the task being performed by the worker from at least the positions of the joints of the worker, and identifies the task being performed by the worker from the result of the inference. A program characterized by.

15. Acquire joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of the worker included in the image; Inferring the task being performed by the worker from the positions of the M joints using one of one or more trained models for inferring the task being performed by the worker from the positions of at least the joints of the worker, and identifying the task being performed by the worker from the result of the inference; When the reliability of the inference is less than a predetermined threshold, N joints (N is an integer satisfying 1≦N<M) are selected from the M joints, and positions of the N joints are used; When the reliability of the inference is less than a predetermined threshold, the inference of the work performed by the worker is re-performed from the target of use by using one of the one or more trained models. A motion analysis method comprising:

16. Acquire joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of the worker included in the image; Obtaining object position information indicating a position of an object included in the image; calculating a distance between the object position and each of the M joint positions; Select N joints (N is an integer satisfying 1≦N<M) from among the joints whose distance is less than a predetermined threshold, and use the positions of the N joints as targets; Using a trained model for inferring the task being performed by a person from the positions of the person's joints and the positions of an object, inferring the task being performed by the worker from the target of use, and identifying the task being performed by the worker from the results of the inference. A motion analysis method comprising:

17. Acquire joint position information indicating positions of M joints (M is an integer equal to or greater than 2) of the worker included in the image; Obtaining object position information indicating a position of an object included in the image; calculating a distance between the object position and each of the M joint positions; In a case where N joints (N is an integer satisfying 1≦N<M) can be selected from the joints whose distance is less than a predetermined threshold, the positions of the N joints selected from the joints whose distance is less than the predetermined threshold are used; When the N joints cannot be selected from the joints whose distance is less than a predetermined threshold, the N joints are selected from predetermined joints, and the positions of the N joints are set as the utilization targets; Inferring the task being performed by the worker from the target of use using one or more trained models for inferring the task being performed by the worker from at least the positions of the joints of the worker, and identifying the task being performed by the worker from the result of the inference. A motion analysis method comprising:

Images