Estimation device, learning device, estimation method, and estimation program

Abstract

The time division unit (11) divides the work engagement time into a work performance time zone, which is a time zone during which the worker is performing any one of the multiple elemental tasks, and a non-work time zone, which is a time zone during which the worker is not performing any one of the multiple elemental tasks, using at least one of the results of a displacement amount determination that determines the amount of displacement of the worker's hand during work engagement time, which is a time zone during which the worker is performing work including multiple elemental tasks, each of which has non-uniform attributes, a tool gripping state determination that determines the tool gripping state of the worker's hand during the work engagement time, and an appearance state determination that determines the appearance state of the worker's hand in the image captured during the work engagement time. The estimation unit (12) estimates the elemental task being performed by the worker during the work performance time zone, based on a partial image that is a portion of the image captured during the work performance time zone.

Description

[Technical field]

[0001] The present disclosure relates to a technique for estimating an elemental work being performed by a worker. An elemental task is an action that can be recognized as a single unit, and is a component of a task. In other words, a task is composed of a combination of multiple elemental tasks. For example, the elemental task "Placing the cover in the installation position" and the elemental task "Screwing the cover in place" are combined to form one task, "Attach the cover." [Background technology]

[0002] Patent Document 1 discloses a technique for capturing an image of a person's behavior and estimating the person's behavior from the captured image. In the technology of Patent Document 1, time-series position data of human body parts obtained from a video is classified into multiple position data. Furthermore, in the technology of Patent Document 1, each position data is analyzed, and a movement sequence (movement, change, rest, etc.) is generated from the analysis result. Then, in the technology of Patent Document 1, a neural network that handles time-series data, such as an RNN (Recurrent Neural Network), analyzes the movement sequence. Furthermore, in the technology of Patent Document 1, a memory neural network processes the analysis result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-22323 Summary of the Invention [Problem to be solved by the invention]

[0004] An operation includes element tasks with various attributes, such as element tasks that take a short time to perform, element tasks that take a long time to perform, element tasks that occur infrequently, and element tasks that occur frequently. When the technology of Patent Document 1 is applied to a task that includes element tasks with such various attributes, the following problems arise.

[0005] Issue (1) The technology of Patent Document 1 uses a neural network that handles time-series data to analyze location data, which is time-series data divided into short intervals. When a neural network is trained on time-series data separated by short intervals for an operation that includes multiple elemental operations with various execution times and / or occurrence frequencies, if the amount of time-series data for each category is large, it becomes difficult to train the network to generate a model for estimating the elemental operations. A category is a label that indicates the type of element work that a worker is performing and a label that indicates a time period during which the worker is not performing any element work. Issue (2) Furthermore, even if a new category of elemental work is created in order to improve the balance of the amount of time-series data between categories, additional data collection is required for the new category. Issue (3) In addition, even if the time series data of an element task that takes a long time to perform is divided into multiple time series data for short periods of time, each time series data only represents a part of the element task. This increases the variability of the time series data even within the same category, making it difficult to learn to generate a model that estimates the element task. Issue (4) Due to the above problems (1) to (3), when an operation includes a plurality of elemental operations whose attributes are not uniform, such as differences in the duration and / or frequency of operation, it is not possible to accurately estimate the elemental operations performed by an operator.

[0006] One of the main objectives of the present disclosure is to solve the above-mentioned problems. More specifically, a main objective of the present disclosure is to enable accurate estimation of elemental work performed by a worker even when the work includes multiple elemental work, each of which has a different attribute. [Means for solving the problem]

[0007] The estimation device according to the present disclosure comprises: a time division unit that divides a work engagement time into a work performance time zone during which the worker is performing any one of the plurality of elemental tasks and a non-work time zone during which the worker is not performing any one of the plurality of elemental tasks, using at least one of a result of a displacement amount determination that determines an amount of displacement of the hand of a worker during a work engagement time that is a time during which the worker is engaged in a task including a plurality of elemental tasks, each of which has non-uniform attributes, while the worker is engaged in the task, a result of a tool gripping state determination that determines a state in which the tool is gripped by the hand of the worker during the work engagement time, and a result of an appearance state determination that determines an appearance state of the hand of the worker in an image captured during the work engagement time; The system further includes an estimation unit that estimates an elemental work being performed by the worker during the work performance time period based on a partial image that is a portion of the image captured during the work performance time period. Effect of the Invention

[0008] According to the present disclosure, even when a task includes a plurality of elemental tasks, each of which has non-uniform attributes, it is possible to accurately estimate the elemental tasks performed by a worker. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing an outline of an example of a functional configuration of an estimation device according to the first embodiment. [Diagram 2] FIG. 4 is a diagram showing an example of time division processing using only the determination result of the amount of displacement according to the first embodiment. [Diagram 3] FIG. 4 is a diagram showing an example of a time division process using only the determination result of the tool gripping state determination according to the first embodiment. [Figure 4] FIG. 13 is a diagram showing an example of time division processing using only the determination result of the appearance situation determination according to the first embodiment. [Diagram 5] FIG. 11 is a diagram showing an example of time division processing using three determination results according to the first embodiment. [Figure 6] 4A to 4C are diagrams showing examples of tool movements during non-work time periods and work execution time periods in the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a detailed functional configuration of the estimation device according to the first embodiment. [Figure 8] 3 is a diagram showing an example of the internal configuration of a task execution time zone detection unit and an element task estimation unit according to the first embodiment. FIG. [Figure 9] 4 is a flowchart showing an example of the operation of the estimation device according to the first embodiment. [Figure 10] 5 is a flowchart showing an example of the operation of an operation execution time period detection unit according to the first embodiment. [Figure 11] 4 is a flowchart showing an example of the operation of an element work estimation unit according to the first embodiment. [Figure 12] FIG. 11 is a diagram showing an example of the functional configuration of a learning device according to a second embodiment. [Figure 13] FIG. 11 is a diagram showing an example of the internal configuration of an element work estimation model generation unit according to the second embodiment. [Figure 14] 10 is a flowchart showing an example of the operation of the learning device according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing an example of the functional configuration of a learning device according to a third embodiment. [Figure 16] FIG. 11 is a diagram showing an example of the internal configuration of an operation execution time period detection model generation unit according to the third embodiment. [Figure 17] 11 is a flowchart showing an example of the operation of the learning device according to the third embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a functional configuration of an estimation device according to a fourth embodiment. [Figure 19] FIG. 13 is a diagram showing an example of the functional configuration of a learning device according to a fifth embodiment. [Figure 20] FIG. 2 is a diagram showing an overview of imaging processing according to the first embodiment. [Figure 21] 4 is a diagram showing an overview of an operation execution time period detection process, an element operation estimation process, and an estimation result process according to the first embodiment. FIG. [Figure 22] FIG. 1 is a diagram showing an example of a hardware configuration of an estimation device according to first and fourth embodiments. [Diagram 23] FIG. 13 is a diagram showing an example of the hardware configuration of a learning device according to the second, third and fifth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described with reference to the drawings. In the following description of the embodiments and in the drawings, the same reference numerals denote the same or corresponding parts.

[0011] Embodiment 1 This disclosure describes a method for accurately estimating the task elements being performed by a worker. An "operator" is a person (worker) or robot (working robot) engaged in work. Additionally, an "operator's hand" includes a human (worker's) hand and a component part of a robot (working robot) that has the same function and / or role as a human hand. Additionally, an "operator's hand" includes at least one of the palm, back of the hand, wrist, and fingers of a human (worker) and a component part of a robot (working robot) that corresponds to any one of the palm, back of the hand, wrist, and fingers. In the following, an example will be described in which a person (worker) is engaged in machine adjustment work.

[0012] ***Configuration Description*** 1 illustrates an example of a functional configuration of an estimation device 100 according to the present embodiment. The estimation device 100 operates in an estimation phase. The estimation device 100 is connected to an imaging device 110 . The image capture device 110 captures an image of the worker's hands during work engagement time. Work engagement time is the time during which the worker is engaged in work. The imaging device 110 outputs an image of the worker's hands obtained by imaging to the estimation device 100. The imaging device 110 is, for example, worn on the head of a worker and captures a first-person perspective image. In this embodiment, the imaging device 110 is assumed to be worn on the head of the worker. However, the imaging device 110 does not have to be worn on the head of the worker as long as it can capture an image of the worker's hands. Furthermore, the image capturing device 110 may not only be a device that captures normal color images, but may also include a sensor having another modality, such as a depth sensor. A depth sensor is capable of capturing images at a wider angle. When the imaging device 110 includes a depth sensor, even if the hand is not captured in the color image, the estimation device 100 may determine that the hand is present as long as the hand is within the imaging range of the depth sensor.

[0013] **Description of a simplified configuration example of the estimation device 100** The estimation device 100 includes a time division unit 11 and an estimation unit 12 . As described later, the estimating device 100 has a detailed functional configuration shown in FIGS. 7 and 8. However, in FIG. 1, the functional configuration of the estimating device 100 is shown in a simplified manner for ease of understanding. Before describing the detailed configuration of the estimation device 100 shown in FIGS. 7 and 8, the functional configuration of the simplified estimation device 100 shown in FIG. 1 will be described.

[0014] **Explanation of time division section 11** The time division unit 11 divides the work engagement time into a work performance time period and a non-work time period using at least one of the results of the displacement amount determination, the results of the tool holding status determination, and the results of the appearance status determination. The displacement amount determination is a process of determining the amount of displacement of the worker's hand. The tool holding state determination is a process for determining the state in which a tool is held by a worker's hand. The tool is a tool used in machine adjustment work. The appearance status determination is a determination process for determining the appearance status of the worker's hands in the video captured during the work engagement time. Further, a work performance time period is a time period during which a worker performs one of a plurality of elemental works. Moreover, a non-working time period is a time period during which the worker is not performing any of the elemental works among a plurality of elemental works. The process performed by the time dividing unit 11 corresponds to a time dividing process.

[0015] **Explanation of Estimation Part 12** The estimation unit 12 estimates the elemental work being performed by the worker during the work performance time period based on a partial image that is a portion of the image captured during the work performance time period that is captured during the work performance time period. Specifically, the estimation unit 12 estimates the element work performed by the worker during the work performance time period, using a trained model generated by training.

[0016] **Time division process explained** 2 to 5 show examples of the time division process performed by the time division unit 11. In FIG. FIG. 2 shows an example in which the time dividing unit 11 divides the work engagement time into a work execution time period and a non-work time period using only the determination result of the amount of displacement determination. FIG. 3 shows an example in which the time dividing unit 11 divides the work engagement time into a work execution time period and a non-work time period using only the result of the tool gripping state determination. FIG. 4 shows an example in which the time dividing unit 11 divides the work engagement time into a work performance time period and a non-work time period using only the determination result of the appearance status determination. FIG. 5 shows an example in which the time division unit 11 divides the work engagement time into a work performance time period and a non-work time period using the results of the displacement amount determination, the results of the tool holding status determination, and the results of the appearance status determination.

[0017] The result of the displacement amount determination in Figure 2 shows the displacement amount of the worker's hand during the work engagement time. In Figure 2, the result of the displacement amount determination is shown in a graph. The time division unit 11 acquires time series data of the joints of the hand derived from an image of the worker's hand captured by the imaging device 110. Then, the amount of displacement of the hand per unit time is determined from the time series data of the joints of the hand. Although not shown in FIG. 1, the estimation device 100 is assumed to include a mechanism for converting an image of the worker's hand into time-series data of the hand joints. The time division unit 11 designates a time period during which the amount of hand displacement per unit time is less than the threshold as a task execution time period, while the time division unit 11 designates a time period during which the amount of hand displacement per unit time is equal to or greater than the threshold as a non-task time period. As shown in (a) of Figure 6, in machine adjustment work, when no work is being performed, the hand holding the tool is thought to move quickly toward (or away from) the work object (this is also thought to be the case for hands not holding a tool). For this reason, time periods when the amount of hand displacement is large are thought to be non-work time periods. On the other hand, as shown in (b) of Figure 6, when work is being performed, the tool is thought to be moved slowly or the range of movement of the tool is small (this is also thought to be the case for hands not holding a tool). For this reason, time periods when the amount of hand displacement is small are thought to be work time periods.

[0018] The result of the tool holding state determination in FIG. 3 indicates whether or not a tool is being held. Although not shown in FIG. 1, the estimation device 100 is assumed to include a mechanism for identifying the tool held by the worker's hand. In the example in Figure 3, first the worker was holding a wrench. Next, the worker was holding a screwdriver. After that, the worker was not holding any tools. Finally, the worker was holding a steel ruler. The time division unit 11 designates a time period during which the worker's hands are holding a tool as a work execution time period, while the time division unit 11 designates a time period during which the worker's hands are not holding any tool as a non-work time period. Furthermore, when the type of tool held by the worker's hand changes over time, the time division unit 11 designates each time period during which the worker's hand holds a different type of tool as a different work performance time period. In the example of Fig. 3, the time division unit 11 designates a time period during which the worker's hand holds a wrench and a time period during which the worker's hand holds a screwdriver as different work performance time periods.

[0019] The determination result of the appearance status determination in FIG. 4 indicates whether or not the worker's hands are appearing in the image captured by the image capturing device 110. The time dividing unit 11 can determine whether or not the worker's hand is present by analyzing the image captured by the imaging device 110 or the time series data of the hand joints. In the example in Figure 4, first the worker's hand appears in the image. Then the worker's hand disappears from the image. Then the worker's hand appears in the image again. The time division unit 11 designates a time period during which the worker's hands appear in the video as a work performance time period, while the time division unit 11 designates a time period during which the worker's hands do not appear in the video as a non-work time period.

[0020] FIG. 5 shows an example in which the time dividing unit 11 divides the work engagement time into a work execution time period and a non-work time period using three determination results. In the example of Figure 5, if the amount of displacement of the worker's hand is equal to or greater than a threshold value, the time division unit 11 designates that time period as a non-work time period, regardless of whether the worker's hand is holding a tool or not and regardless of whether the worker's hand appears in the image or not. Furthermore, when the amount of displacement of the worker's hand is less than the threshold, if the worker's hand appears in the video, the time division unit 11 designates the time period as a work execution time period regardless of whether the worker's hand is holding a tool or not. On the other hand, even if the amount of displacement of the worker's hand is less than the threshold, if the worker's hand does not appear in the video, the time division unit 11 designates the time period as a non-work time period regardless of whether the worker's hand is holding a tool or not. In this manner, when two or more of the determination results of the displacement amount determination, the determination result of the tool holding status determination, and the determination result of the appearance status determination are used, the user of the estimation device 100 defines in advance application criteria for determining the priority order in which the two or more determination results are to be applied.

[0021] The estimation unit 12 applies the time-series data of the hand joints derived from the partial images captured during the work performance time period to the learned model, and estimates the elemental work performed by the worker during the work performance time period. The trained model used by the estimation unit 12 is generated by a training device 200 described later.

[0022] In this way, the estimation device 100 according to this embodiment performs a two-stage process of detecting a task execution time period by the time dividing unit 11 and estimating an element task using a partial video by the estimation unit 12. Therefore, the estimating device 100 according to this embodiment can accurately estimate the work elements performed by the workers.

[0023] **Explanation of an Example of the Hardware Configuration of the Estimation Device 100** The estimation device 100 according to this embodiment is a computer having a hardware configuration exemplified in FIG. As shown in FIG. 22, the estimation device 100 includes, as hardware, a processor 801, a main storage device 802, an auxiliary storage device 803, and a communication device 804. The functions of the time dividing unit 11 and the estimation unit 12 shown in FIG. 1, and further the components shown in FIG. 7 and FIG. 8, are realized by, for example, a program. The auxiliary storage device 803 stores programs that realize the functions of these components. These programs are loaded from the auxiliary storage device 803 to the main storage device 802. Then, the processor 801 executes these programs to perform the operations of these components.

[0024] The operation procedure of the estimation device 100 corresponds to an estimation method, and the program that realizes the operation of the estimation device 100 corresponds to an estimation program.

[0025] **Explanation of a detailed configuration example of the estimation device 100** FIG. 7 illustrates an example of a detailed functional configuration of the estimation device 100 illustrated in FIG. In FIG. 7 , the estimation device 100 includes a joint position time-series data acquisition unit 120, a joint velocity calculation unit 121, a joint time-series data imaging unit 122, an task performance time zone detection unit 123, an element work estimation unit 124, an estimation result processing unit 125, an element work estimation model storage unit 130, a pre-processing statistics storage unit 131, and an estimation result storage unit 132. The task execution time zone detection unit 123 corresponds to the time division unit 11 shown in Fig. 1. Furthermore, the element task estimation unit 124 corresponds to the estimation unit 12 shown in Fig. 1.

[0026] **Explanation of joint position time series data acquisition unit 120** The joint position time-series data acquisition unit 120 acquires an image V obtained by capturing an image of a worker's hand from the imaging device 110. The image V is an image captured during work time in the estimation phase. The image V includes frames in which the worker's hand is captured and frames in which the worker's hand is not captured. The joint position time series data acquisition unit 120 generates joint position time series data HPT from the video V. The joint position time series data HPT is data indicating the spatial position coordinates of each joint of the hand. The joint position time-series data acquisition unit 120 generates the joint position time-series data HPT, for example, by using an existing hand tracking model that estimates the positions of the joints of the hand. The joint position time series data acquisition unit 120 generates right hand joint position time series data HPT and left hand joint position time series data HPT. Note that, hereinafter, when simply referring to the joint position time series data HPT, it refers to both the right hand joint position time series data HPT and the left hand joint position time series data HPT. When there is a period of time in which no hand appears in the video V, the joint position time series data acquisition unit 120 inserts a value that enables determination that no hand appears into the joint position time series data HPT. Then, the joint position time series data acquiring unit 120 outputs the joint position time series data HPT to the joint velocity calculating unit 121 and the joint time series data imaging unit 122. In addition, the joint position time series data acquiring unit 120 may output the joint position time series data HPT to the task execution time zone detecting unit 123.

[0027] **Explanation of joint velocity calculation part 121** The joint velocity calculation unit 121 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120 . Then, the joint velocity calculation unit 121 calculates the difference (velocity) in the time direction for each joint of the joint position time series data HPT. Then, the joint velocity calculation unit 121 outputs the joint velocity time series data HVT indicating the calculation result to the joint time series data imaging unit 122. The joint velocity calculation unit 121 may output the joint velocity time series data HVT to the task execution time zone detection unit 123. The joint velocity calculation unit 121 generates the joint velocity time series data HVT of the right hand and the joint velocity time series data HVT of the left hand. Note that, hereinafter, when simply referring to the joint velocity time series data HVT, it refers to both the joint velocity time series data HVT of the right hand and the joint velocity time series data HVT of the left hand.

[0028] **Explanation of joint time series data imaging unit 122** The joint time series data imaging unit 122 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120. In addition, the joint time series data imaging unit 122 acquires the joint velocity time series data HVT from the joint velocity calculation unit 121. The joint time series data imaging unit 122 images the joint position time series data HPT to generate a left hand joint position image LJPI and a right hand joint position image RJPI, and also images the joint velocity time series data HVT to generate a left hand joint velocity image LJVI and a right hand joint velocity image RJVI. Then, the joint time-series data visualization unit 122 outputs the left hand joint position image LJPI, the right hand joint position image RJPI, the left hand joint velocity image LJVI, and the right hand joint velocity image RJVI to the task execution time zone detection unit 123.

[0029] Here, imaging of the joint position time series data HPT and imaging of the joint velocity time series data HVT by the joint time series data imaging unit 122 will be described. 20 shows an overview of the imaging process of the joint time-series data imaging unit 122. Here, the imaging process of the joint position time-series data HPT will be described. Imaging refers to generating time-series data of the three-dimensional coordinates of a joint by regarding the x, y, and z values ​​of the three-dimensional coordinates as R, G, and B of an image. 20, the joint position time-series data HPT for each time t includes spatial position coordinates of J (J≧2) joints. The coordinate axes C are the x-axis, the y-axis, and the z-axis.

[0030] The joint time-series data imaging unit 122 generates a tensor having elements each of the J joints j and the coordinate value c of each axis of the left hand joint position time-series data HPT, for example. Then, the joint time-series data imaging unit 122 combines the tensors in the time direction to generate a left hand joint position image LJPI. The joint time-series data imaging unit 122 also generates the right wrist joint position image RJPI in a similar procedure. Further, the joint time-series data imaging unit 122 generates a left hand joint velocity image LJVI from the left hand joint velocity time-series data HVT in a similar procedure. Further, the joint time-series data imaging unit 122 generates a right hand joint velocity image RJVI from the right hand joint velocity time-series data HVT in a similar procedure.

[0031] **Explanation of the gripping tool time series data generator 126** The gripping tool time-series data generator 126 acquires the video V or the sensor information RRS. The video V is an image captured by the imaging device 110. In other words, the gripping tool time-series data generating unit 126 may acquire the video V from the joint position time-series data acquiring unit 120, or may acquire the video V from the imaging device 110 independently of the joint position time-series data acquiring unit 120. The sensor information RRS is information that indicates whether the worker is holding a tool or not, and if the worker is holding a tool, the sensor information RRS indicates the type of tool that the worker is holding. For example, when a tag-type wireless communication sensor such as an RFID is attached to the hand of a worker, the tag-type wireless communication sensor receives a signal transmitted from the tool held by the worker. The tag-type wireless communication sensor then determines the type of tool held by the worker based on the received signal. Furthermore, when the tag-type wireless communication sensor does not receive a signal, it determines that the worker is not holding a tool. The tag-type wireless communication sensor outputs the determination result to the held tool time-series data generator 126 as sensor information RRS.

[0032] The gripped tool time-series data generating unit 126 uses the video V or the sensor information RRS to identify whether the worker's hand is gripping a tool. If the worker's hand is gripping a tool, the gripped tool time-series data generating unit 126 identifies the type of tool the worker's hand is gripping. When acquiring the video V, the held tool time-series data generation unit 126 identifies from each frame of the video V whether the worker is holding a tool and the type of tool, for example, by using a known object detection model. When acquiring the sensor information RRS, the held tool time-series data generator 126 analyzes the sensor information RRS to identify whether or not the worker is holding a tool and the type of tool.

[0033] The gripped tool time-series data generating unit 126 outputs the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 indicating the identification result to the task execution time zone detecting unit 123. The left hand held tool data LTO1 is data that indicates whether or not the worker's left hand is holding a tool and the type of tool being held in a time series. In other words, by analyzing the left hand held tool data LTO1, it is possible to identify the time period when the worker's left hand is holding a tool and the time period when the worker's left hand is not holding a tool. In addition, by analyzing the left hand held tool data LTO1, it is possible to identify the type of tool being held in the worker's left hand in a time period when the worker's left hand is holding a tool in a time series. Similarly, the right hand held tool data RTO1 is data that indicates whether the worker's right hand is holding a tool and the type of tool being held in a time series. In other words, by analyzing the right hand held tool data RTO1, it is possible to identify the time period when the worker's right hand is holding a tool and the time period when the worker's right hand is not holding a tool in a time series. In addition, by analyzing the right hand held tool data RTO1, it is possible to identify the type of tool being held by the worker's right hand during the time period when the worker's right hand is holding a tool.

[0034] **Explanation of the work execution time zone detection unit 123** The task execution time zone detection unit 123 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time series data visualization unit 122. Furthermore, the task execution time zone detection unit 123 acquires the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 from the gripped tool time series data generation unit 126.

[0035] The task performance time zone detection unit 123 analyzes the left hand joint position image LJPI and / or the left hand joint velocity image LJVI. Then, the task performance time zone detection unit 123 generates a left hand appearance time zone set LS, which is a set of time zones in which the left hand appears in the video. Similarly, the task performance time zone detection unit 123 analyzes the right hand joint position image RJPI and / or the right hand joint velocity image RJVI. Then, the task performance time zone detection unit 123 generates a right hand appearance time zone set RS, which is a set of time zones in which the right hand appears in the image. In this manner, the operation execution time zone detection unit 123 performs the occurrence status determination shown in FIG. Furthermore, when the task performance time zone detection unit 123 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120, the task performance time zone detection unit 123 may determine the appearance status using the joint position time series data HPT. When the task performance time zone detection unit 123 acquires the joint velocity time series data HVT from the joint velocity calculation unit 121, the task performance time zone detection unit 123 may determine the appearance status using the joint velocity time series data HVT.

[0036] Furthermore, the work performance time zone detection unit 123 extracts, in chronological order, time zones during which a tool is being held and time zones during which a tool is not being held for each of the left and right hands based on the left hand held tool data LTO1 and the right hand held tool data RTO1. Furthermore, when the type of tool held by the worker is changing, the work performance time zone detection unit 123 extracts different time zones for each type of tool. Hereinafter, the extraction result of the work execution time zone detection unit 123 is referred to as a tool gripping state determination result TS. The tool holding status determination result TS indicates, in chronological order, the time periods when the worker is holding the tool and the time periods when the worker is not holding the tool. The tool holding status determination result TS also associates the time periods when the worker is holding the tool with the type of tool the worker is holding in that time period. In this manner, the task execution time zone detection unit 123 performs the tool holding state determination shown in FIG.

[0037] Furthermore, the task execution time zone detection unit 123 analyzes the left hand joint position image LJPI and / or the left hand joint velocity image LJVI, and calculates the amount of displacement of the left hand per unit time. Similarly, the task execution time zone detection unit 123 analyzes the right hand joint position image RJPI and / or the right hand joint velocity image RJVI to calculate the amount of displacement of the right hand per unit time. In this manner, the work execution time zone detection unit 123 performs the displacement amount determination of FIG. Furthermore, when the task performance time zone detection unit 123 acquires the joint position time series data HPT from the joint position time series data acquisition unit 120, the task performance time zone detection unit 123 may determine the amount of displacement using the joint position time series data HPT. When the task performance time zone detection unit 123 acquires the joint velocity time series data HVT from the joint velocity calculation unit 121, the task performance time zone detection unit 123 may determine the amount of displacement using the joint velocity time series data HVT.

[0038] The work performance time zone detection unit 123 divides the work performance time zone into a work performance time zone and a non-work time zone based on at least one of the determination results. Specifically, the work performance time zone detection unit 123 identifies the start time and end time of the work performance time zone, and identifies the start time and end time of the non-work time zone, and divides the work performance time zone into a work performance time zone and a non-work time zone. In addition, when the work performance time zone detection unit 123 divides the work engagement time into a work performance time zone and a non-work time zone using only the result of the appearance status determination, it is not necessary to perform the tool gripping status determination and the displacement amount determination. Similarly, when the work execution time zone detection unit 123 divides the work engagement time into a work execution time zone and a non-work time zone using only the result of the tool gripping status determination, it does not need to perform the appearance status determination and the displacement amount determination. Similarly, when the work performance time zone detection unit 123 divides the work engagement time into a work performance time zone and a non-work time zone using only the result of the displacement amount determination, it does not need to perform the appearance status determination and the tool holding status determination.

[0039] The work performance time zone detection unit 123 outputs a work performance time zone set FS, which is a set of work performance time zones, to the element work estimation unit 124. The work performance time zone set FS indicates the start time and end time of the work performance time zone for each work performance time zone. Furthermore, the work execution time zone detection unit 123 outputs a non-work time zone set NFS, which is a set of non-work time zones, to the estimation result processing unit 125. The non-work time zone set NFS indicates the start time and end time of each non-work time zone.

[0040] **Explanation of element work estimation unit 124** The element work estimation unit 124 acquires the work performance time zone set FS from the work performance time zone detection unit 123. Furthermore, the element work estimation unit 124 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging unit 122. Furthermore, the element work estimation unit 124 acquires the element work estimation model M from the element work estimation model storage unit 130. The element work estimation model M corresponds to the trained model shown in FIG. Furthermore, the element work estimation unit 124 acquires preprocessing statistics from the estimation result storage unit 132. Specifically, the element work estimation unit 124 acquires, as the preprocessing statistics, a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS.

[0041] The joint position luminance average value PM is the average luminance value of the joint position image for training obtained by the training device 200 described later. The joint position luminance standard deviation PS is the standard deviation of the luminance of the joint position images for training obtained by the training device 200. The brightness is the coordinate value for each coordinate axis C included in the joint position image. The learning joint position images are joint position images used in the learning phase, and correspond to the left hand joint position image LJPI and right hand joint position image RJPI described above.

[0042] Strictly speaking, the average joint position luminance value PM includes the average left hand joint position luminance value LPM and the average right hand joint position luminance value RPM. The average left hand joint position luminance value LPM is the average value obtained from the left hand joint position label image sLJPI, which is the left hand joint position image for learning. The average right hand joint position luminance value RPM is the average value obtained from the right hand joint position label image sRJPI, which is the right hand joint position image for learning. Moreover, the left hand joint position luminance average value LPM is obtained for each coordinate axis C. For this reason, the left hand joint position luminance average value LPM can be expressed strictly as the left hand joint position luminance average value LPM(C). Similarly, the right hand joint position luminance average value RPM is also obtained for each coordinate axis C. Therefore, in a strict expression, the right hand joint position luminance average value RPM is expressed as the right hand joint position luminance average value RPM(C). Unless a strict notation is required, the left hand joint position luminance average value LPM and the right hand joint position luminance average value RPM are collectively referred to as the joint position luminance average value PM. Note that, instead of using the average luminance value for the left hand and the average luminance value for the right hand as described above, the average luminance value for both the left hand and the right hand may be used. In the following, an example in which the average luminance value for the left hand and the average luminance value for the right hand are used will be described, but the following description also applies to the case in which the average luminance value for both the left hand and the right hand is used.

[0043] Strictly speaking, the joint position luminance standard deviation PS also includes the left hand joint position luminance standard deviation LPS and the right hand joint position luminance standard deviation RPS. The left hand joint position luminance standard deviation LPS is the standard deviation obtained from the left hand joint position label image sLJPI, which is the left hand joint position image for training. The right hand joint position luminance standard deviation RPS is the standard deviation obtained from the right hand joint position label image sRJPI, which is the right hand joint position image for training. In addition, the left hand joint position luminance standard deviation LPS is also found for each coordinate axis C. For this reason, the left hand joint position luminance standard deviation LPS can be expressed strictly as the left hand joint position luminance standard deviation LPS(C). Similarly, the right hand joint position luminance standard deviation RPS is also found for each coordinate axis C. Therefore, in a strict sense, the right hand joint position luminance standard deviation RPS is expressed as the right hand joint position luminance standard deviation RPS(C). Unless a strict notation is required, the left hand joint position luminance standard deviation LPS and the right hand joint position luminance standard deviation RPS are collectively referred to as the joint position luminance standard deviation PS. Note that, instead of using the luminance standard deviation for the left hand and the luminance standard deviation for the right hand as described above, the luminance standard deviation over both the left hand and the right hand may be used. In the following, an example in which the luminance standard deviation for the left hand and the luminance standard deviation for the right hand are used will be described, but the following description also applies to the case in which the luminance standard deviation over both the left hand and the right hand is used.

[0044] The joint velocity luminance average value VM is the average luminance value of the joint velocity image for training obtained by the training device 200. The joint velocity luminance standard deviation VS is the standard deviation of the luminance of the training joint velocity image obtained by the training device 200. The learning joint velocity images are joint velocity images used in the learning phase, and correspond to the left hand joint velocity image LJVI and right hand joint velocity image RJVI described above.

[0045] Strictly speaking, the joint velocity brightness average value VM also includes the left hand joint velocity brightness average value LVM and the right hand joint velocity brightness average value RVM. The left hand joint velocity brightness average value LVM is the average value obtained from the left hand joint velocity labeled image sLJVI, which is the left hand joint velocity image for learning. The right hand joint velocity brightness average value RVM is the average value obtained from the right hand joint velocity labeled image sRJVI, which is the right hand joint velocity image for learning. In addition, the left wrist joint velocity luminance average value LVM is also obtained for each coordinate axis C. Therefore, in a strict expression, the left wrist joint velocity luminance average value LVM is expressed as the left wrist joint velocity luminance average value LVM(C). Similarly, the right hand joint velocity brightness average value RVM is also obtained for each coordinate axis C. Therefore, in a strict sense, the right hand joint velocity brightness average value RVM is expressed as the right hand joint velocity brightness average value RVM(C). Unless strictly required, the left wrist joint velocity brightness average value LVM and the right wrist joint velocity brightness average value RVM are collectively referred to as the joint velocity brightness average value VM. Note that, instead of using the average luminance value for the left hand and the average luminance value for the right hand as described above, the average luminance value for both the left hand and the right hand may be used. In the following, an example in which the average luminance value for the left hand and the average luminance value for the right hand are used will be described, but the following description also applies to the case in which the average luminance value for both the left hand and the right hand is used.

[0046] Strictly speaking, the joint velocity brightness standard deviation VS also includes the left hand joint velocity brightness standard deviation LVS and the right hand joint velocity brightness standard deviation RVS. The left hand joint velocity brightness standard deviation LVS is the standard deviation obtained from the left hand joint velocity label image sLJVI, which is the left hand joint velocity image for learning. The right hand joint velocity brightness standard deviation RVS is the standard deviation obtained from the right hand joint velocity label image sRJVI, which is the right hand joint velocity image for learning. In addition, the left wrist joint velocity luminance standard deviation LVS is also calculated for each coordinate axis C. Therefore, the left wrist joint velocity luminance standard deviation LVS can be expressed strictly as the left wrist joint velocity luminance standard deviation LVS(C). Similarly, the right wrist joint velocity luminance standard deviation RVS is also calculated for each coordinate axis C. Therefore, in a strict expression, the right wrist joint velocity luminance standard deviation RVS is expressed as the right wrist joint velocity luminance standard deviation RVS(C). Unless a strict notation is required, the left hand joint velocity brightness standard deviation LVS and the right hand joint velocity brightness standard deviation RVS are collectively referred to as the joint velocity brightness standard deviation VS. Note that, instead of using the luminance standard deviation for the left hand and the luminance standard deviation for the right hand as described above, the luminance standard deviation over both the left hand and the right hand may be used. In the following, an example in which the luminance standard deviation for the left hand and the luminance standard deviation for the right hand are used will be described, but the following description also applies to the case in which the luminance standard deviation over both the left hand and the right hand is used.

[0047] The element work estimation unit 124 extracts partial images constituting a partial video captured during a work performance time period from each of the left hand joint position image LJPI, the left hand joint velocity image LJVI, the right hand joint position image RJPI, and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS. That is, the element work estimation unit 124 extracts images constituting a partial video captured during a work performance time period from the left hand joint position image LJPI as partial images. The element work estimation unit 124 also extracts similar partial images from the left hand joint velocity image LJVI, the right hand joint position image RJPI, and the right hand joint velocity image RJV. In the following, a set of partial images extracted from each of the left hand joint position image LJPI, left hand joint velocity image LJVI, right hand joint position image RJPI, and right hand joint velocity image RJVI will be referred to as a partial image set IS.

[0048] Then, the element work estimation unit 124 resizes each partial image included in the partial image set IS for each work time period into an image having a certain width and height. Furthermore, the element work estimation unit 124 uses the pre-processing statistics to standardize the pixel values ​​included in each partial image after resizing for each coordinate axis C. Note that the order of resizing and standardization may be reversed. Specifically, the element work estimation unit 124 standardizes the pixel values ​​of the resized partial image extracted from the left hand joint position image LJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS. In addition, the element work estimation unit 124 standardizes the pixel values ​​of the resized partial image extracted from the right hand joint position image RJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS. More specifically, the element work estimation unit 124 standardizes the pixel values ​​of the resized partial image extracted from the left hand joint position image LJPI using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C). In addition, the element work estimation unit 124 standardizes the pixel values ​​of the resized partial image extracted from the right hand joint position image RJPI, using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C).

[0049] Furthermore, the element work estimation unit 124 standardizes the pixel values ​​of the resized partial image extracted from the left hand joint velocity image LJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Furthermore, the element work estimation unit 124 standardizes the pixel values ​​of the resized partial image extracted from the right hand joint velocity image RJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. More specifically, the element work estimation unit 124 standardizes the pixel values ​​of the resized partial image extracted from the left hand joint velocity image LJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C). In addition, the element work estimation unit 124 standardizes the pixels of the resized partial image extracted from the right hand joint velocity image RJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C).

[0050] Then, the element work estimation unit 124 inputs each partial image after resizing and standardization to the element work estimation model M. Then, the element work estimation unit 124 acquires the estimation result qs from the element work estimation model M. The estimation result qs indicates, in association with a work time period, an element work that is estimated to be performed by a worker during that work time period. The element work estimation unit 124 performs the above procedure for all work performance time periods included in the work performance time period set FS, and obtains estimation results qs for each work time period. Then, the element work estimation unit 124 outputs an estimation result set QS, which is a set of the estimation results qs for each work time period, to the estimation result processing unit 125.

[0051] **Explanation of the estimation result processing unit 125** The estimation result processing unit 125 acquires the estimation result set QS from the element work estimation unit 124. In addition, the estimation result processing unit 125 acquires the non-work time slot set NFS from the work execution time slot detection unit 123. The estimation result processing unit 125 sorts the multiple estimation results qs included in the estimation result set QS in ascending order of the work implementation time zones to which each estimation result qs corresponds. The estimation result processing unit 125 also assigns a label representing a non-work time slot to each of the non-work time slots included in the non-work time slot set NFS. Furthermore, the estimation result processing unit 125 inserts each non-work time slot after the label assignment into the corresponding time position in the sorted estimation result set QS. Each non-work time slot after labeling and the sorted estimation result set QS are output to the estimation result storage unit 132 as the final estimation result AS.

[0052] FIG. 21 shows an overview of the operations of the work execution time zone detection unit 123, the element work estimation unit 124, and the estimation result processing unit 125 described above.

[0053] **Explanation of the element work estimation model storage unit 130** The element work estimation model storage unit 130 stores the element work estimation model M.

[0054] **Explanation of preprocessing statistics storage unit 131** The pre-processing statistics storage unit 131 stores the pre-processing statistics (joint position luminance average value PM, joint position luminance standard deviation PS, joint velocity luminance average value VM, and joint velocity luminance standard deviation VS).

[0055] **Explanation of the estimation result storage unit 132** The estimation result storage unit 132 stores the final estimation result AS.

[0056] **Explanation of an example of the internal configuration of the work execution time zone detection unit 123 and the internal configuration of the element work estimation unit 124** FIG. 8 shows an example of the internal configuration of the work execution time zone detection unit 123 and an example of the internal configuration of the element work estimation unit 124. An example of the internal configuration of the work execution time zone detection unit 123 and an example of the internal configuration of the element work estimation unit 124 will be described with reference to FIG. In FIG. 8, of the functional components of the estimation device 100, only the functional components necessary for explaining an example of the internal configuration of the work execution time zone detection unit 123 and the example of the internal configuration of the element work estimation unit 124 are shown.

[0057] **Explanation of an example of the internal configuration of the work execution time period detection unit 123** The task execution time zone detection unit 123 has, as its internal components, an appearance status determination unit 1231, a tool gripping status determination unit 1232, a displacement amount determination unit 1233, and a task execution time zone determination unit 1234.

[0058] **Explanation of the appearance status determination unit 1231** The appearance status determination unit 1231 performs appearance status determination. The appearance status determination unit 1231 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging unit 122. Then, the appearance status determination unit 1231 analyzes the left hand joint position image LJPI and / or the left hand joint velocity image LJVI to generate a left hand appearance time period set LS. Similarly, the appearance status determination unit 1231 analyzes the right hand joint position image RJPI and / or the right hand joint velocity image RJVI to generate a right hand appearance time period set RS. As described above, the left hand appearance time period set LS is a set of time periods when the left hand appears in the video. Also, the right hand appearance time period set RS is a set of time periods when the right hand appears in the video. The left hand appearance time period set LS and the right hand appearance time period set RS are time series data shown in FIG. 4 as the determination result of the appearance status determination. The appearance status determining unit 1231 outputs the left hand appearance time period set LS and the right hand appearance time period set RS to the work implementation time period determining unit 1234 . As described above, the appearance status determining unit 1231 may perform appearance status determination using the joint position time series data HPT and / or the joint velocity time series data HVT. In FIG. 8, the input of the joint position time series data HPT and / or the joint velocity time series data HVT to the appearance situation determination unit 1231 is omitted.

[0059] **Description of the tool holding status determination unit 1232** The tool holding state determining unit 1232 determines the tool holding state. The tool holding state determination unit 1232 acquires the left hand held tool data LTO1 and the right hand held tool data RTO1 from the tool holding time-series data generation unit 126. Then, the tool holding status determination unit 1232 extracts, in chronological order, the time periods during which a tool is being held and the time periods during which a tool is not being held for each of the left and right hands based on the left hand held tool data LTO1 and the right hand held tool data RTO1. In addition, when the type of tool held by the worker is changing, the tool holding status determination unit 1232 treats the time periods during which a tool is being held as different time periods for each type of tool. The tool gripping situation determination unit 1232 outputs a left hand tool gripping situation determination result LTS indicating the extraction result for the left hand to the work performance time zone determination unit 1234. In addition, the tool gripping situation determination unit 1232 outputs a right hand tool gripping situation determination result RTS indicating the extraction result for the right hand to the work performance time zone determination unit 1234. The left hand tool gripping situation determination result LTS and the right hand tool gripping situation determination result RTS are time series data shown in Fig. 4 as the determination results of the tool gripping situation determination.

[0060] **Explanation of the displacement amount determination unit 1233** The displacement amount determination unit 1233 performs displacement amount determination. The displacement amount determination unit 1233 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging unit 122. Then, the displacement amount determination unit 1233 analyzes the left hand joint position image LJPI and / or the left hand joint velocity image LJVI to calculate the displacement amount of the left hand per unit time. For example, the displacement amount determination unit 1233 calculates the displacement amount of the wrist joint of the left hand per unit time. Similarly, the displacement amount determination unit 1233 analyzes the right hand joint position image RJPI and / or the right hand joint velocity image RJVI to calculate the displacement amount of the right hand per unit time. For example, the displacement amount determination unit 1233 calculates the displacement amount of the wrist joint of the right hand per unit time. Then, the displacement amount determination unit 1233 outputs left hand displacement amount data LZS indicating the displacement amount of the left hand per unit time in a time series to the work execution time zone determination unit 1234. The displacement amount determination unit 1233 also outputs right hand displacement amount data RZS indicating the displacement amount of the right hand per unit time in a time series to the work execution time zone determination unit 1234. The left hand displacement amount data LZS and right hand displacement amount data RZS are time series data shown in Fig. 2 as the determination result of the displacement amount determination. As described above, the displacement amount determining unit 1233 may perform the displacement amount determination using the joint position time series data HPT and / or the joint velocity time series data HVT. In FIG. 8, the input of the joint position time series data HPT and / or the joint velocity time series data HVT to the displacement amount determination unit 1233 is omitted.

[0061] **Explanation of the work execution time zone determination unit 1234** The operation implementation time zone determination unit 1234 acquires the left hand appearance time zone set LS and the right hand appearance time zone set RS from the appearance status determination unit 1231. In addition, the task execution time zone determination unit 1234 acquires the left hand tool gripping situation determination result LTS and the right hand tool gripping situation determination result RTS from the tool gripping situation determination unit 1232. Furthermore, the task execution time zone determination unit 1234 acquires the left hand displacement amount data LZS and the right hand displacement amount data RZS from the displacement amount determination unit 1233. Then, the work execution time zone determining unit 1234 divides the work engagement time into a work execution time zone and a non-work time zone based on these.

[0062] For example, if the displacement amount of at least one of the left hand and the right hand is equal to or greater than a threshold value, the work performance time zone determination unit 1234 designates the time zone as a non-work time zone, regardless of whether at least one of the left hand and the right hand is holding a tool or whether at least one of the left hand and the right hand appears in the image.

[0063] Furthermore, for example, when the displacement amounts of the left and right hands are both less than the threshold, if at least one of the left and right hands appears in the image, the task execution time zone determination unit 1234 designates the time zone as a task execution time zone, regardless of whether at least one of the left and right hands is holding a tool. On the other hand, even if the displacement amounts of the left and right hands are both less than the threshold, if neither the left nor right hand appears in the image, the time division unit 11 designates the time zone as a non-work time zone, regardless of whether at least one of the left and right hands is holding a tool. In this case, the work execution time zone determination unit 1234 may treat the left hand and the right hand separately. Specifically, the work execution time zone determination unit 1234 treats the following (A1) to (A3) as separate work execution time zones. Also, when the work engagement time is divided into a work execution time zone and a non-work time zone only by the appearance status determination, the work execution time zone determination unit 1234 treats (A1) to (A3) as separate work execution time zones. (A1) The time period when the left hand appears but the right hand does not appear (A2) The time period when the left hand is not appearing but the right hand is appearing (A3) Time period when both the left and right hands are present

[0064] Furthermore, for example, when at least one of the left hand and the right hand is holding a tool, the task performance time zone determination unit 1234 designates that time zone as the task performance time zone, regardless of the amount of displacement of the left hand and the right hand. In this case, the work execution time zone determination unit 1234 may treat the left hand and the right hand separately. Specifically, the work execution time zone determination unit 1234 treats the following (B1) to (B3) as separate work execution time zones. Also, when dividing the work engagement time into a work execution time zone and a non-work time zone only by the tool gripping state determination, the work execution time zone determination unit 1234 treats (B1) to (B3) as separate work execution time zones. (B1) The time period when the left hand is holding a tool but the right hand is not holding a tool (B2) The time period when the left hand is not holding a tool, but the right hand is holding a tool. (B3) Time period when both the left and right hands are holding the tool Furthermore, the work execution time period determining unit 1234 may subdivide the above (B3) and treat the following (B3-1) and (B3-2) as separate work execution time periods. (B3-1) Time period when the left and right hands are holding the same tool (B3-2) Time period when the left and right hands are holding different tools

[0065] As described above, the work execution time zone determination unit 1234 designates a time zone in which the amount of displacement is less than the threshold as the work execution time zone. The task execution time zone determining unit 1234 may compare the amount of displacement with a threshold value, for example, as follows. The task execution time zone determination unit 1234 regards the left hand appearance time zone set LS as a set of task execution time zone candidates. The task execution time zone determination unit 1234 also extracts time series data of partial displacement amounts of time zones of each candidate in the left hand displacement amount data LZS. This time series data of partial displacement amounts is called a left hand partial displacement amount time series data set sLZS. The i-th data of the left hand partial displacement amount time series data set sLZS is also called sLZS(i). The task execution time zone determination unit 1234 divides sLZS(i) into a certain time interval, and the k-th partial displacement amount data of the divided sLZS(i) is called short-time partial displacement amount data sLZS(i,k). The task execution time zone determination unit 1234 calculates statistics of the amount of variation contained in the short-time partial variation amount data sLZS(i,k). The statistics is, for example, an average value. If the statistic is equal to or greater than the threshold, the work execution time zone determination unit 1234 determines that the time zone corresponding to the short-time partial variation data sLZS(i,k) is a non-work execution time zone. On the other hand, if the statistic is less than the threshold, the work execution time zone determination unit 1234 determines that the time zone corresponding to the short-time partial variation data sLZS(i,k) is a work execution time zone. The task execution time zone determination unit 1234 performs similar processing on the right hand appearance time zone RS and the right hand tool gripping state determination result RTS. Furthermore, when a left hand and a right hand appear at the same time, one of which is holding a tool and the other is not, the task execution time zone determination unit 1234 may determine whether or not it is a task execution time zone based on the amount of displacement of the hand holding the tool. Alternatively, the task execution time zone determination unit 1234 may determine whether or not it is a task execution time zone based on a comparison between a threshold value and a statistical amount of displacement of the hand holding the tool obtained by the process described in this paragraph. When both hands are holding a tool, the task execution time zone determination unit 1234 compares the displacement amount of the left hand with the displacement amount of the right hand, or compares the statistics of the displacement amount of the left hand obtained by the processing described in this paragraph with the statistics of the displacement amount of the right hand. Then, the task execution time zone determination unit 1234 may compare the smaller of the displacement amount or the statistics of the displacement amount with a threshold value to determine whether or not it is a task execution time zone.

[0066] The work execution time zone determination unit 1234 may divide the work engagement time into work execution time zones and non-work time zones based on criteria other than the above.

[0067] The work implementation time slot determination unit 1234 outputs a work implementation time slot set FS, which is a set of work implementation time slots, to the element work estimation unit 124. The work implementation time slot set FS indicates the start time and end time of the work implementation time slot for each work implementation time slot. Furthermore, the work implementation time slot determination unit 1234 outputs a non-work time slot set NFS, which is a set of non-work time slots, to the estimation result processing unit 125. The non-work time slot set NFS indicates the start time and end time of each non-work time slot.

[0068] **Explanation of an example of the internal configuration of the element work estimation unit 124** The element work estimation unit 124 has, as its internal components, a joint position image acquisition unit 1241, a joint velocity image acquisition unit 1242, a joint position feature extraction unit 1243, a joint velocity feature extraction unit 1244, and a joint image feature classification unit 1245.

[0069] **Explanation of joint position image acquisition unit 1241** The joint position image acquisition unit 1241 acquires the task execution time period set FS from the task execution time period determination unit 1234 . Furthermore, the joint position image acquisition section 1241 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging section 122. Furthermore, the joint position image acquisition unit 1241 acquires preprocessing statistics from the preprocessing statistics storage unit 131. Specifically, the joint position image acquisition unit 1241 acquires a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS as the preprocessing statistics.

[0070] Then, the joint position image acquisition unit 1241 extracts partial images constituting a partial video captured during the work performance time period from each of the left hand joint position image LJPI and the right hand joint position image RJPI for each work performance time period included in the work performance time period set FS. Then, the joint position image acquisition unit 1241 resizes each partial image to an image having a certain width and height for each work time period. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values ​​of the resized partial image extracted from the left hand joint position image LJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values ​​of the resized partial image extracted from the right hand joint position image RJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. As described above, the order of resizing and standardization may be reversed. The joint position image acquisition unit 1241 outputs the resized and standardized partial images of each of the left hand joint position image LJPI and the right hand joint position image RJPI to the joint position feature extraction unit 1243.

[0071] In addition, the joint position image acquisition section 1241 outputs the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average value VM, and the joint velocity brightness standard deviation VS to the joint velocity image acquisition section 1242. Here, an example is described in which the joint position image acquisition unit 1241 acquires a task execution time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS, and outputs these values ​​to the joint velocity image acquisition unit 1242. Alternatively, the joint velocity image acquisition unit 1242 may acquire the work performance time period set FS from the work performance time period determination unit 1234, acquire the left hand joint velocity image LJVI and the right hand joint velocity image RJVI from the joint time series data imaging unit 122, and acquire the joint velocity brightness average value VM and the joint velocity brightness standard deviation VS from the pre-processing statistics storage unit 131. In this case, the joint position image acquisition unit 1241 acquires only the left hand joint position image LJPI and the right hand joint position image RJPI from the joint time series data imaging unit 122, and acquires only the joint position luminance average value PM and the joint position luminance standard deviation PS from the preprocessing statistics storage unit 131.

[0072] **Explanation of the joint velocity image acquisition unit 1242** The joint velocity image acquisition unit 1242 acquires from the joint position image acquisition unit 1241 a task execution time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS. Then, the joint velocity image acquisition unit 1242 extracts partial images constituting a partial video captured during the work performance time period from each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS. Then, the joint velocity image acquisition unit 1242 resizes each partial image to an image with a fixed width and height for each task time period. Furthermore, the joint velocity image acquisition unit 1242 standardizes pixel values ​​of the resized partial image extracted from the left hand joint velocity image LJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Furthermore, the joint velocity image acquisition unit 1242 standardizes pixel values ​​of the resized partial image extracted from the right hand joint velocity image RJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. The joint velocity image acquisition unit 1242 outputs the resized and standardized partial images of each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI to the joint velocity feature extraction unit 1244 .

[0073] **Explanation of the joint position feature extraction unit 1243** The joint position feature extraction unit 1243 acquires from the joint position image acquisition unit 1241 partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI. Then, the joint position feature extraction unit 1243 inputs the acquired partial image to the element work estimation model M. The element work estimation model M used here is, for example, a pre-trained convolutional neural network. The joint position feature extraction unit 1243 extracts a joint position feature vector, which is a feature vector, from each partial image. Here, the joint position feature vector obtained from the partial image of the left hand joint position image LJPI is called the left hand position feature vector fLP. Also, the joint position feature vector obtained from the partial image of the right hand joint position image RJPI is called the right hand position feature vector fRP. The joint position feature extraction unit 1243 outputs the left hand position feature vector fLP and the right hand position feature vector fRP to the joint image feature classification unit 1245.

[0074] **Explanation of the joint velocity feature extraction unit 1244** The joint velocity feature extraction unit 1244 acquires from the joint velocity image acquisition unit 1242 partial images after resizing and standardization for each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI. Then, the joint velocity feature extraction unit 1244 inputs the acquired partial image to the element work estimation model M. The element work estimation model M used here is also, for example, a pre-trained convolutional neural network. Note that the convolutional neural network used here may be the same as or different from the convolutional neural network used by the joint position feature extraction unit 1243. The joint velocity feature extraction unit 1244 extracts a joint velocity feature vector, which is a feature vector, from each partial image. Here, the joint velocity feature vector obtained from the partial image for the left hand joint velocity image LJVI is called the left hand velocity feature vector fLV. Also, the joint velocity feature vector obtained from the partial image for the right hand joint velocity image RJVI is called the right hand velocity feature vector fRV. The joint velocity feature extraction unit 1244 outputs the left hand velocity feature vector fLV and the right hand velocity feature vector fRV to the joint image feature classification unit 1245.

[0075] **Explanation of the joint image feature classification unit 1245** The joint image feature classification unit 1245 acquires the left hand position feature vector fLP and the right hand position feature vector fRP from the joint position feature extraction unit 1243. In addition, the joint image feature classification unit 1245 acquires the left hand velocity feature vector fLV and the right hand velocity feature vector fRV from the joint velocity feature extraction unit 1244.

[0076] Then, the joint image feature classifying unit 1245 combines the left hand position feature vector fLP, the right hand position feature vector fRP, the left hand velocity feature vector fLV, and the right hand velocity feature vector fRV. Furthermore, the joint image feature classification unit 1245 inputs the combined feature vector to the element work estimation model M. More specifically, the joint image feature classification unit 1245 inputs the combined feature vector to a layer of a neural network that performs classification processing of the element work estimation model M. The layer configuration of the element work estimation model M can be arbitrarily specified by the user of the estimation device 100 depending on the amount of data and the task. The element work estimation model M is generally configured using multiple fully connected layers and multiple activation layers. The joint image feature classification unit 1245 estimates, for each task performance time period, the component task with the highest probability among the component tasks estimated by the component task estimation model M as the component task being performed by the worker during that task time period. Then, the joint image feature classifying unit 1245 outputs an estimation result set QS, which is a set of the estimation results qs for each work time period, to the estimation result processing unit 125.

[0077] ***Explanation of Operation*** Next, an example of the operation of the estimation device 100 according to the present embodiment will be described. FIG. 9 is a flowchart showing an example of the operation of the estimation device 100.

[0078] First, in step S11, the joint position time series data acquisition unit 120 acquires the joint position time series data HPT. That is, the joint position time series data acquisition unit 120 generates the joint position time series data HPT from the video V from the imaging device 110.

[0079] Next, in step S12, the joint velocity calculation unit 121 generates joint velocity time series data HVT from the joint position time series data HPT.

[0080] Next, in step S13, the joint time series data imaging unit 122 images the joint position time series data HPT to generate a left hand joint position image LJPI and a right hand joint position image RJPI. Also, the joint time series data imaging unit 122 images the joint velocity time series data HVT to generate a left hand joint velocity image LJVI and a right hand joint velocity image RJVI.

[0081] Next, in step S14, the gripped tool time-series data generator 126 generates left hand gripped tool data LTO1 and right hand gripped tool data RTO1 using the video V or the sensor information RRS.

[0082] Next, in step S15, the work performance time zone detection unit 123 detects the work performance time zone using the left hand joint position image LJPI, the right hand joint position image RJPI, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the left hand held tool data LTO1, and the right hand held tool data RTO1.

[0083] Next, in step S16, the element work estimation unit 124 uses the element work estimation model M to estimate the element work to be performed by the worker for each work performance time period.

[0084] Finally, in step S17, the estimation result processing unit 125 outputs the final estimation result AS.

[0085] Next, an example of the operation of the work execution time zone detection unit 123 will be described in detail. FIG. 10 is a flowchart showing details of an example of the operation of the work execution time zone detection unit 123.

[0086] First, in step S111, the appearance status determination unit 1231 determines the appearance status. The appearance status determination unit 1231 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging unit 122. Then, the appearance status determination unit 1231 analyzes the left hand joint position image LJPI and / or the left hand joint velocity image LJVI to generate a left hand appearance time period set LS. Similarly, the appearance status determination unit 1231 analyzes the right hand joint position image RJPI and / or the right hand joint velocity image RJVI to generate a right hand appearance time period set RS. The appearance status determining unit 1231 outputs the left hand appearance time period set LS and the right hand appearance time period set RS to the work implementation time period determining unit 1234.

[0087] Next, in step S112, the tool holding state determining unit 1232 determines the tool holding state. The tool holding state determination unit 1232 acquires the left hand held tool data LTO1 and the right hand held tool data RTO1 from the tool holding time-series data generation unit 126. Then, the tool holding status determination unit 1232 extracts, in chronological order, the time periods during which a tool is being held and the time periods during which a tool is not being held for each of the left and right hands based on the left hand held tool data LTO1 and the right hand held tool data RTO1. In addition, when the type of tool held by the worker is changing, the tool holding status determination unit 1232 treats the time periods during which a tool is being held as different time periods for each type of tool. The tool gripping situation determination unit 1232 outputs to the task execution time zone determination unit 1234 a left hand tool gripping situation determination result LTS indicating the extraction result for the left hand and a right hand tool gripping situation determination result RTS indicating the extraction result for the right hand.

[0088] Next, in step S113, the displacement amount determination unit 1233 determines the displacement amount. The displacement amount determination unit 1233 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging unit 122. Then, the displacement amount determination unit 1233 analyzes the left hand joint position image LJPI and / or the left hand joint velocity image LJVI to calculate the displacement amount of the left hand per unit time. Similarly, the displacement amount determination unit 1233 analyzes the right hand joint position image RJPI and / or the right hand joint velocity image RJVI to calculate the displacement amount of the right hand per unit time. Then, the displacement amount determination unit 1233 outputs left hand displacement amount data LZS that indicates the displacement amount of the left hand per unit time in a time series, and right hand displacement amount data RZS that indicates the displacement amount of the right hand per unit time in a time series, to the work execution time zone determination unit 1234.

[0089] Finally, in step S114, the work execution time zone determination unit 1234 detects the work execution time zone. The operation implementation time zone determination unit 1234 acquires the left hand appearance time zone set LS and the right hand appearance time zone set RS from the appearance status determination unit 1231. In addition, the task execution time zone determination unit 1234 acquires the left hand tool gripping situation determination result LTS and the right hand tool gripping situation determination result RTS from the tool gripping situation determination unit 1232. Furthermore, the task execution time zone determination unit 1234 acquires the left hand displacement amount data LZS and the right hand displacement amount data RZS from the displacement amount determination unit 1233. Then, the work execution time zone determining unit 1234 divides the work engagement time into a work execution time zone and a non-work time zone based on these. Finally, the task performance time zone determination unit 1234 outputs a task performance time zone set FS, which is a set of task performance time zones, to the joint image feature classification unit 1245. In addition, the work implementation time slot determination unit 1234 outputs a non-work time slot set NFS, which is a set of non-work time slots, to the estimation result processing unit 125.

[0090] Next, an example of the operation of the element work estimation unit 124 will be described in detail. FIG. 11 is a flowchart showing details of an example of the operation of the element work estimation unit 124.

[0091] First, in step S121, the joint position image acquisition unit 1241 generates a partial image of the joint position image, and resizes and standardizes the partial image. The joint position image acquisition unit 1241 acquires the task execution time period set FS from the task execution time period determination unit 1234 . Furthermore, the joint position image acquisition section 1241 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging section 122. In addition, the joint position image acquisition unit 1241 acquires the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS as preprocessing statistics from the preprocessing statistics storage unit 131.

[0092] Then, the joint position image acquisition unit 1241 extracts partial images constituting a partial video captured during the work performance time period from each of the left hand joint position image LJPI and the right hand joint position image RJPI for each work performance time period included in the work performance time period set FS. Then, the joint position image acquisition unit 1241 resizes each partial image to an image having a certain width and height for each work time period. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values ​​of the resized partial image extracted from the left hand joint position image LJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. Furthermore, the joint position image acquisition unit 1241 standardizes the pixel values ​​of the resized partial image extracted from the right hand joint position image RJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. As described above, the order of resizing and standardization may be reversed. The joint position image acquisition unit 1241 outputs the resized and standardized partial images of each of the left hand joint position image LJPI and the right hand joint position image RJPI to the joint position feature extraction unit 1243. In addition, the joint position image acquisition section 1241 outputs the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average value VM, and the joint velocity brightness standard deviation VS to the joint velocity image acquisition section 1242.

[0093] Next, in step S122, the joint velocity image acquisition section 1242 generates a partial image of the joint velocity image, and resizes and standardizes the partial image. The joint velocity image acquisition unit 1242 acquires from the joint position image acquisition unit 1241 a task execution time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS. Then, the joint velocity image acquisition unit 1242 extracts partial images constituting a partial video captured during the work performance time period from each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS. Then, the joint velocity image acquisition unit 1242 resizes each partial image to an image with a fixed width and height for each task time period. Furthermore, the joint velocity image acquisition unit 1242 standardizes pixel values ​​of the resized partial image extracted from the left hand joint velocity image LJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Furthermore, the joint velocity image acquisition unit 1242 standardizes pixel values ​​of the resized partial image extracted from the right hand joint velocity image RJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. As described above, the order of resizing and standardization may be reversed. The joint velocity image acquisition unit 1242 outputs the resized and standardized partial images of each of the left wrist joint velocity image LJVI and the right wrist joint velocity image RJVI to the joint velocity feature extraction unit 1244 .

[0094] Next, in step S123, the joint position feature extracting unit 1243 extracts a joint position feature vector. The joint position feature extraction unit 1243 acquires from the joint position image acquisition unit 1241 partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI. Then, the joint position feature extraction unit 1243 inputs the acquired partial image to the element work estimation model M. Then, the joint position feature extraction unit 1243 extracts a left hand position feature vector fLP and a right hand position feature vector fRP. The joint position feature extraction unit 1243 outputs the left hand position feature vector fLP and the right hand position feature vector fRP to the joint image feature classification unit 1245.

[0095] Next, in step S124, the joint velocity feature extractor 1244 extracts a joint velocity feature vector. The joint velocity feature extraction unit 1244 acquires from the joint velocity image acquisition unit 1242 partial images after resizing and standardization for each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI. Then, the joint velocity feature extraction unit 1244 inputs the acquired partial image to the element task estimation model M. Then, the joint velocity feature extraction unit 1244 extracts a left hand velocity feature vector fLV and a right hand velocity feature vector fRV. The joint velocity feature extraction unit 1244 outputs the left hand velocity feature vector fLV and the right hand velocity feature vector fRV to the joint image feature classification unit 1245.

[0096] Finally, in step S125, the joint image feature classifier 1245 combines the joint position feature vector and the joint velocity feature vector to estimate an element work. The joint image feature classification unit 1245 acquires the left hand position feature vector fLP and the right hand position feature vector fRP from the joint position feature extraction unit 1243. In addition, the joint image feature classification unit 1245 acquires the left hand velocity feature vector fLV and the right hand velocity feature vector fRV from the joint velocity feature extraction unit 1244. Then, the joint image feature classification unit 1245 combines the left hand position feature vector fLP, the right hand position feature vector fRP, the left hand velocity feature vector fLV, and the right hand velocity feature vector fRV. Furthermore, the joint image feature classification unit 1245 inputs the combined feature vector to the element task estimation model M. The joint image feature classification unit 1245 estimates, for each task performance time period, the component task with the highest probability among the component tasks estimated by the component task estimation model M as the component task being performed by the worker during that task time period. Then, the joint image feature classifying unit 1245 outputs an estimation result set QS, which is a set of the estimation results qs for each work time period, to the estimation result processing unit 125.

[0097] ***Description of the Effects of the Embodiment*** In this embodiment, the estimation device 100 performs a two-stage process of detecting an operation execution time period and estimating an element operation being performed during the detected operation execution time period. In detecting the task performance time period, the estimation device 100 detects the task performance time period by analyzing the appearance of the hand, the gripping state of the tool, and the amount of displacement of the hand. Therefore, the estimation device 100 can accurately detect the task performance time period even if the occurrence frequency and / or execution time of elemental tasks included in a task vary. Since the work performance time period can be accurately detected in this manner, in estimating the component work, the estimation device 100 can accurately estimate the component work being performed during the work performance time period by using the partial image corresponding to the work performance time period. Therefore, according to this embodiment, even if a task includes a plurality of elemental tasks each having different attributes, it is possible to accurately estimate the elemental tasks performed by a worker.

[0098] In addition, in this embodiment, the estimation device 100 estimates the element work by using an image recognition model. Therefore, according to this embodiment, an existing trained model can be used to estimate the element work. In other words, since there is no need to train a model from scratch, the amount of data required for training can be reduced, and the burden of data collection can be reduced.

[0099] Embodiment 2 In this embodiment, a learning device that generates the element work estimation model M described in the first embodiment will be described. In this embodiment, differences from the first embodiment will be mainly described. It should be noted that matters not explained below are the same as those in the first embodiment.

[0100] **Configuration Description** 12 illustrates an example of a functional configuration of the learning device 200 according to this embodiment. The learning device 200 operates in a learning phase. The learning phase is a phase preceding the estimation phase in which the estimation device 100 according to the first embodiment operates. The learning device 200 is connected to an imaging device 210 . The imaging device 210 is similar to the imaging device 110 described in the first embodiment. That is, the imaging device 210 outputs an image of the worker's hands to the learning device 200. The imaging device 210 is, for example, worn on the head of the worker and captures video from a first-person perspective. In this embodiment, the imaging device 210 is assumed to be worn on the head of the worker. However, the imaging device 210 does not have to be worn on the head of the worker as long as it can capture video of the worker's hands.

[0101] **Explanation of an example of the hardware configuration of the learning device 200** The learning device 200 according to this embodiment is a computer having a hardware configuration exemplified in FIG. As shown in FIG. 23, the learning device 200 includes, as hardware, a processor 901, a main memory device 902, an auxiliary memory device 903, and a communication device 904. The functions of the components such as the joint position time-series data acquisition unit 220 and the joint velocity calculation unit 221 shown in FIG. 12 are realized by, for example, a program. The auxiliary storage device 903 stores programs that realize the functions of these components. These programs are loaded from the auxiliary storage device 903 to the main storage device 902. Then, the processor 901 executes these programs to perform the operations of these components.

[0102] **Explanation of an example of the functional configuration of the learning device 200** FIG. 13 shows an example of the functional configuration of the learning device 200. The learning device 200 includes a joint position time-series data acquisition unit 220, a joint velocity calculation unit 221, a joint time-series data imaging unit 222, a learning data generation unit 223, a preprocessing statistics calculation unit 224, an element work estimation model generation unit 225, a learning data storage unit 230, a preprocessing statistics storage unit 231 and an element work estimation model storage unit 232.

[0103] **Explanation of joint position time series data acquisition unit 220** The joint position time-series data acquiring section 220 performs the same operations as the joint position time-series data acquiring section 120 described in the first embodiment. That is, the joint position time-series data acquisition unit 220 acquires an image V obtained by capturing an image of the worker's hand from the imaging device 210. The image V is an image captured during the work engagement time in the learning phase. The image V includes frames in which the worker's hand is captured and frames in which the worker's hand is not captured. The joint position time series data acquisition unit 220 generates joint position time series data HPT from the video V. Then, the joint position time series data acquisition unit 220 outputs the joint position time series data HPT to the joint velocity calculation unit 221 and the joint time series data imaging unit 222. The joint position time-series data HPT according to this embodiment is similar to the joint position time-series data HPT described in the first embodiment.

[0104] **Explanation of joint velocity calculation unit 221** The joint velocity calculation unit 221 performs the same operation as the joint velocity calculation unit 121 described in the first embodiment. The joint velocity calculation unit 221 acquires the joint position time series data HPT from the joint position time series data acquisition unit 220 . Then, the joint velocity calculation unit 221 calculates the difference (velocity) in the time direction for each joint of the joint position time series data HPT. Then, the joint velocity calculation unit 221 outputs the joint velocity time series data HVT indicating the calculation result to the joint time series data visualization unit 222. The joint velocity time series data HVT according to this embodiment is similar to the joint velocity time series data HVT described in the first embodiment.

[0105] **Explanation of joint time-series data imaging unit 222** The joint time-series data imaging unit 222 performs the same operation as the joint time-series data imaging unit 122 described in the first embodiment. That is, the joint time series data imaging unit 222 acquires the joint position time series data HPT from the joint position time series data acquisition unit 220. In addition, the joint time series data imaging unit 222 acquires the joint velocity time series data HVT from the joint velocity calculation unit 221. The joint time series data imaging unit 222 images the joint position time series data HPT to generate a left hand joint position image LJPI and a right hand joint position image RJPI. The joint time series data imaging unit 122 images the joint velocity time series data HVT to generate a left hand joint velocity image LJVI and a right hand joint velocity image RJVI. Then, the joint time-series data visualization unit 222 outputs the left hand joint position image LJPI, the right hand joint position image RJPI, the left hand joint velocity image LJVI, and the right hand joint velocity image RJVI to the learning data generation unit 223. The left hand joint position image LJPI and right hand joint position image RJPI and the left hand joint velocity image LJVI and right hand joint velocity image RJVI in this embodiment are similar to the left hand joint position image LJPI and right hand joint position image RJPI and the left hand joint velocity image LJVI and right hand joint velocity image RJVI described in embodiment 1.

[0106] **Explanation of the learning data generation unit 223** The learning data generation unit 223 acquires a left hand joint position image LJPI, a right hand joint position image RJPI, a left hand joint velocity image LJVI, and a right hand joint velocity image RJVI from the joint time-series data visualization unit 222. In addition, the learning data generation unit 223 acquires the label information LBLinf from, for example, the user of the learning device 200. The label information LBLinf includes multiple labels LBL. Each label LBL includes a start time ts, an end time te, and an element task type typ. Here, the user of the learning device 200 manually associates the start time ts and the end time te with the element task type typ. However, the start time ts and the end time te with the element task type typ may be associated in other ways. The start time ts indicates the start time of the work execution time period. The end time te indicates the start time of the work execution time period. The element work type typ indicates the type of element work being performed during the work performance time period.

[0107] The learning data generation unit 223 extracts an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te as a left hand joint position label image sLJPI for each label LBL of the label information LBLinf from the left hand joint position image LJPI. Then, the learning data generation unit 223 associates the extracted left hand joint position label image sLJPI with the label LBL. In addition, the learning data generation unit 223 extracts an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te as a right hand joint position label image sRJPI for each label LBL of the label information LBLinf from the right hand joint position image RJPI. Then, the learning data generation unit 223 associates the extracted right hand joint position label image sRJPI with the label LBL. Also, from the left hand joint velocity image LJVI, for each label LBL in the label information LBLinf, an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te is extracted as a left hand joint velocity label image sLJVI. Then, the learning data generation unit 223 associates the extracted left hand joint velocity label image sLJVI with the label LBL. Furthermore, the learning data generation unit 223 extracts, from the right hand joint velocity image RJVI, an image region from an image position corresponding to the start time ts to an image position corresponding to the end time te for each label LBL in the label information LBLinf, as a right hand joint velocity label image sRJVI. Then, the learning data generation unit 223 associates the extracted right hand joint velocity label image sRJVI with the label LBL. Furthermore, the learning data generation unit 223 stores multiple pairs of left hand joint position label images sLJPI and labels LBL, multiple pairs of right hand joint position label images sRJPI and labels LBL, multiple pairs of left hand joint velocity label images sLJVI and labels LBL, and multiple pairs of right hand joint velocity label images sRJVI and labels LBL in the learning data storage unit 230 as learning data sIs.

[0108] **Explanation of pre-processing statistics calculation unit 224** The pre-processing statistics calculation unit 224 obtains the learning data sIs from the learning data storage unit 230 . Specifically, the preprocessing statistics calculation unit 224 acquires, as the learning data sIs, multiple pairs of left hand joint position label images sLJPI and labels LBL, multiple pairs of right hand joint position label images sRJPI and labels LBL, multiple pairs of left hand joint velocity label images sLJVI and labels LBL, and multiple pairs of right hand joint velocity label images sRJVI and labels LBL.

[0109] Then, the pre-processing statistics calculation unit 224 calculates, from the learning data sIs, the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS as pre-processing statistics. Specifically, the preprocessing statistics calculation unit 224 calculates the average luminance value in the multiple left hand joint position label images sLJPI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint position label image sLJPI is the left hand joint position luminance average value LPM(C). In addition, the preprocessing statistics calculation unit 224 calculates the average luminance value in the multiple right hand joint position label images sRJPI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint position label image sRJPI is the right hand joint position luminance average value RPM(C). Furthermore, the preprocessing statistics calculation unit 224 calculates the standard deviation of luminance in the multiple left hand joint position label images sLJPI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint position label image sLJPI is the left hand joint position luminance standard deviation LPS(C). In addition, the preprocessing statistics calculation unit 224 calculates the standard deviation of luminance in multiple right hand joint position label images sRJPI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint position label image sRJPI is the right hand joint position luminance standard deviation RPS(C). Note that, instead of using the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand as described above, the average luminance value and standard luminance deviation for both the left hand and the right hand may be used. In the following, an example will be described in which the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand are used, but the following description also applies to the case in which the average luminance value and standard luminance deviation for both the left hand and the right hand are used.

[0110] In addition, the preprocessing statistics calculation unit 224 calculates the average brightness value in the multiple left hand joint velocity label images sLJVI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint velocity label image sLJVI is the left hand joint velocity brightness average value LVM(C). In addition, the preprocessing statistics calculation unit 224 calculates the average brightness value in multiple right hand joint velocity label images sRJVI included in the learning data sIs for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint velocity label image sRJVI is the right hand joint velocity brightness average value RVM(C). Furthermore, the preprocessing statistics calculation unit 224 calculates the standard deviation of brightness in the multiple left hand joint velocity label images sLJVI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the left hand joint velocity label image sLJVI is the left hand joint velocity brightness standard deviation LVS(C). In addition, the preprocessing statistics calculation unit 224 calculates the standard deviation of brightness in multiple right hand joint velocity label images sRJVI included in the learning data sIs for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 224 from the right hand joint velocity label image sRJVI is the right hand joint velocity brightness standard deviation RVS(C). Note that, instead of using the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand as described above, the average luminance value and standard luminance deviation for both the left hand and the right hand may be used. In the following, an example will be described in which the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand are used, but the following description also applies to the case in which the average luminance value and standard luminance deviation for both the left hand and the right hand are used.

[0111] The pre-processing statistics calculation unit 224 stores the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS in the pre-processing statistics storage unit 231 .

[0112] **Explanation of the element work estimation model generation unit 225** The element work estimation model generation unit 225 acquires learning data sIs from the learning data storage unit 230. Specifically, the element work estimation model generation unit 225 acquires, as the learning data sIs, a plurality of pairs of left hand joint position label images sLJPI and labels LBL, a plurality of pairs of right hand joint position label images sRJPI and labels LBL, a plurality of pairs of left hand joint velocity label images sLJVI and labels LBL, and a plurality of pairs of right hand joint velocity label images sRJVI and labels LBL. In addition, the element work estimation model generation unit 225 acquires the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS from the preprocessing statistics storage unit 231.

[0113] Next, the element work estimation model generation unit 225 resizes the left hand joint position label image sLJPI included in the learning data sIS into an image having a certain width and height. Furthermore, the element work estimation model generation unit 225 standardizes the pixel values ​​of the resized left hand joint position label image sLJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS. Specifically, the element work estimation model generation unit 225 standardizes the pixel values ​​of the resized left hand joint position label image sLJPI using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C).

[0114] Furthermore, the element work estimation model generation unit 225 resizes the right hand joint position label image sRJPI included in the learning data sIS into an image having a certain width and height. Furthermore, the element work estimation model generation unit 225 standardizes the pixel values ​​of the resized right hand joint position label image sRJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS. Specifically, the element task estimation model generation unit 225 standardizes the pixel values ​​of the resized right hand joint position labeled image sRJPI using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C).

[0115] Furthermore, the element work estimation model generation unit 225 resizes the left hand joint velocity label image sLJVI included in the learning data sIS into an image having a certain width and height. Furthermore, the element work estimation model generation unit 225 standardizes the pixel values ​​of the resized left hand joint velocity labeled image sLJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Specifically, the element task estimation model generation unit 225 standardizes the pixel values ​​of the resized left hand joint velocity labeled image sLJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C).

[0116] Furthermore, the element work estimation model generation unit 225 resizes the right hand joint velocity label image sRJVI included in the learning data sIS into an image having a certain width and height. Furthermore, the element work estimation model generation unit 225 standardizes the pixel values ​​of the resized right hand joint velocity labeled image sRJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Specifically, the element task estimation model generation unit 225 standardizes the pixel values ​​of the resized right hand joint velocity labeled image sRJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C).

[0117] Next, the element work estimation model generation unit 225 inputs each of the resized and standardized label images to a predefined neural network. The label images to be input to the neural network are the resized and standardized left hand joint position label image sLJPI, the resized and standardized right hand joint position label image sRJPI, the resized and standardized left hand joint velocity label image sLJVI, and the resized and standardized right hand joint velocity label image sRJVI. The neural network is a combination of a pre-trained convolutional neural network and a fully connected layer. The number of layers and the type of layers to be used in the pre-trained convolutional neural network are arbitrarily specified by the user of the learning device 200.

[0118] The element task estimation model generation unit 225 acquires a feature map of each label image extracted by the convolutional neural network. The feature map of the left hand joint position label image sLJPI after resizing and standardization is called the feature map fLP. The feature map of the right hand joint position label image sRJPI after resizing and standardization is called the feature map fRP. The feature map of the left hand joint velocity label image sLJVI after resizing and standardization is called the feature map fLV. The feature map of the right wrist joint velocity label image sRJVI after resizing and standardization is called the feature map fRV.

[0119] The element work estimation model generation unit 225 vectorizes and combines these feature maps fLP, fRP, fLV, and fRV. Furthermore, the element work estimation model generation unit 225 inputs the combined feature map to a neural network, learns the weights of the neural network, and generates an element work estimation model M. Then, the element work estimation model generation unit 225 stores the element work estimation model M in the element work estimation model storage unit 232.

[0120] **Explanation of the learning data storage unit 230** The learning data storage unit 230 stores the learning data sIs.

[0121] **Explanation of preprocessing statistics storage unit 231** The pre-processing statistics storage unit 231 stores the pre-processing statistics (joint position luminance average value PM, joint position luminance standard deviation PS, joint velocity luminance average value VM, and joint velocity luminance standard deviation VS).

[0122] **Explanation of the element work estimation model storage unit 232** The element work estimation model storage unit 232 stores the element work estimation model M.

[0123] **Explanation of an example of the internal configuration of the element work estimation model generation unit 225** FIG. 13 shows an example of the internal configuration of the element work estimation model generation unit 225. An example of the internal configuration of the element work estimation model generation unit 225 will be described with reference to FIG. In FIG. 13, of the functional components of the learning device 200, only the functional components necessary for explaining an example of the internal configuration of the element work estimation model generation unit 225 are shown.

[0124] The element work estimation model generation unit 225 has, as its internal components, a joint position image acquisition unit 2251, a joint velocity image acquisition unit 2252, a joint position feature extraction unit 2253, a joint velocity feature extraction unit 2254, and a joint feature classification learning unit 2255. The joint position image acquisition unit 2251 corresponds to the joint position image acquisition unit 1241 shown in FIG. The joint velocity image acquisition section 2252 corresponds to the joint velocity image acquisition section 1242 shown in FIG. The joint position feature extraction unit 2253 corresponds to the joint position feature extraction unit 1243 shown in FIG. The joint velocity feature extractor 2254 corresponds to the joint velocity feature extractor 1244 shown in FIG. The joint image feature classification learning unit 2255 corresponds to the joint image feature classification unit 1245 shown in FIG.

[0125] **Explanation of joint position image acquisition unit 2251** The joint position image acquisition unit 2251 acquires the learning data sIs from the learning data storage unit 230. Specifically, the joint position image acquisition unit 2251 acquires, as the learning data sIs, multiple pairs of left hand joint position label images sLJPI and labels LBL, multiple pairs of right hand joint position label images sRJPI and labels LBL, multiple pairs of left hand joint velocity label images sLJVI and labels LBL, and multiple pairs of right hand joint velocity label images sRJVI and labels LBL. Furthermore, the joint position image acquisition unit 2251 acquires preprocessing statistics from the preprocessing statistics storage unit 231. Specifically, the joint position image acquisition unit 2251 acquires, as the preprocessing statistics, a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS.

[0126] Then, the joint position image acquisition unit 2251 resizes the left hand joint position label image sLJPI to an image having a certain width and height. Furthermore, the joint position image acquisition unit 2251 standardizes the pixel values ​​of the resized left hand joint position label image sLJPI for each coordinate axis C using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C). In addition, the joint position image acquisition unit 2251 resizes the right hand joint position label image sRJPI to an image with a certain width and height. Furthermore, the joint position image acquisition unit 2251 standardizes the pixel values ​​of the resized right hand joint position label image sRJPI using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C) for each coordinate axis C. The joint position image acquisition unit 2251 outputs the resized and standardized left hand joint position label image sLJPI and the resized and standardized right hand joint position label image sRJPI to the joint position feature extraction unit 2253. Note that, instead of using the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand as described above, the average luminance value and standard luminance deviation for both the left hand and the right hand may be used. In the following, an example will be described in which the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand are used, but the following description also applies to the case in which the average luminance value and standard luminance deviation for both the left hand and the right hand are used.

[0127] In addition, the joint position image acquisition unit 2251 outputs the left hand joint velocity labeled image sLJVI, the right hand joint velocity labeled image sRJVI, the joint velocity brightness average value VM, and the joint velocity brightness standard deviation VS to the joint velocity image acquisition unit 2252. Here, an example is described in which the joint position image acquisition unit 2251 acquires a left hand joint velocity label image sLJVI, a right hand joint velocity label image sRJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS, and outputs these to the joint velocity image acquisition unit 2252. Alternatively, the joint velocity image acquisition unit 2252 may acquire the left hand joint velocity label image sLJVI and the right hand joint velocity label image sRJVI from the learning data storage unit 230, and acquire the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS from the preprocessing statistics storage unit 231. In this case, the joint position image acquisition unit 2251 acquires only the left hand joint position label image sLJPI and the right hand joint position label image sRJPI from the learning data storage unit 230, and acquires only the joint position luminance average value PM and the joint position luminance standard deviation PS from the preprocessing statistics storage unit 231.

[0128] **Explanation of the joint velocity image acquisition unit 2252** The joint velocity image acquisition unit 2252 acquires a left hand joint velocity labeled image sLJVI, a right hand joint velocity labeled image sRJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS from the joint position image acquisition unit 2251. Then, the joint velocity image acquisition unit 2252 resizes the left hand joint velocity labeled image sLJVI to an image with a certain width and height. Furthermore, the joint velocity image acquisition unit 2252 standardizes the pixel values ​​of the resized left hand joint velocity labeled image sLJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C) for each coordinate axis C. The joint velocity image acquisition unit 2252 also resizes the right hand joint velocity labeled image sRJPI to an image with a certain width and height. Furthermore, the joint velocity image acquisition unit 2252 standardizes the pixel values ​​of the resized right hand joint velocity labeled image sRJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C) for each coordinate axis C. The joint velocity image acquisition unit 2252 outputs the resized and standardized left hand joint velocity label image sLJVI and the resized and standardized right hand joint velocity label image sRJVI to the joint velocity feature extraction unit 2254.

[0129] **Explanation of the joint position feature extraction unit 2253** The joint position feature extraction unit 2253 acquires a resized and standardized left hand joint position label image sLJPI and a resized and standardized right hand joint position label image sRJPI from the joint position image acquisition unit 2251. Then, the joint position feature extraction unit 2253 inputs the resized and standardized left hand joint position label image sLJPI to a pre-trained convolutional neural network to obtain a feature vector fLP.

[0130] The joint position feature extraction unit 2253 may also perform network learning, i.e., calculation of the weights of the network, based on the loss value propagated from the joint image feature classification unit 2255 described later. For example, the weights of a pre-trained network may be used as initial values ​​to update the weights of all layers. For calculating the weights, the joint position feature extraction unit 2253 uses, for example, the commonly used backpropagation algorithm. Furthermore, the joint position feature extraction unit 2253 may use other known methods to calculate the weights.

[0131] As a learning process, the joint position feature extraction unit 2253 repeats acquisition of a feature map fLP, calculation of a loss value L, and calculation of a weight a predetermined number of times. Finally, the joint position feature extraction unit 2253 obtains a feature map fLP of the left hand joint position label image sLJPI after resizing and standardization. In addition, the joint position feature extraction unit 2253 performs similar processing on the right hand joint position label image sRJPI after resizing and standardization, and obtains the feature map fRP of the right hand joint position label image sRJPI after resizing and standardization. The joint position feature extraction unit 2253 outputs the feature map fLP and the feature map fRP to the joint image feature classification learning unit 2255.

[0132] **Explanation of the joint velocity feature extraction unit 2254** The joint velocity feature extraction unit 2254 acquires a resized and standardized left hand joint velocity label image sLJVI and a resized and standardized right hand joint velocity label image sRJVI from the joint velocity image acquisition unit 2252. Then, the joint velocity feature extraction unit 2254 performs the same process as the joint position feature extraction unit 2253 to obtain a feature map fLV of the left hand joint velocity label image sLJVI after resizing and standardization. Also, the joint velocity feature extraction unit 2254 obtains a feature map fRV of the right hand joint velocity label image sRJVI after resizing and standardization. Then, the joint velocity feature extraction unit 2254 outputs the feature map fLV and the feature map fRV to the joint image feature classification learning unit 2255.

[0133] **Explanation of the joint image feature classification learning unit 2255** The joint image feature classification learning unit 2255 acquires the feature maps fLP and fRP from the joint position feature extraction unit 2253. In addition, the joint image feature classification learning unit 2255 acquires the feature maps fLV and fRV from the joint velocity feature extraction unit 2254. Then, the joint image feature classification learning unit 2255 vectorizes and combines the feature maps fLP, fRP, fLV, and fRV. Furthermore, the joint image feature classification learning unit 2255 inputs the combined feature map to a neural network, learns the weights of the neural network, and generates an element task estimation model M. The joint image feature classification learning unit 2255 uses, for example, cross-entropy, which is generally used for classification learning, to calculate the loss value L. The joint position feature extraction unit 2253 may use a loss value corresponding to imbalance data, such as focal loss, so as to deal with variations in the amount of data depending on the type of label LBL. In addition, the joint image feature classification learning unit 2255 may use a loss function that enables distance learning based on the relationship between the angle between feature vectors and the element task type typ of the label LBL, so as to deal with the variation in data. Then, the joint image feature classification learning unit 2255 stores the element work estimation model M in the element work estimation model storage unit 232.

[0134] ***Explanation of Operation*** Next, an example of the operation of learning device 200 according to this embodiment will be described. FIG. 14 is a flowchart showing an example of the operation of the learning device 200.

[0135] First, in step S21, the joint position time-series data acquisition unit 220 generates joint position time-series data HPT from the video V from the imaging device 210.

[0136] Next, in step S22, the joint velocity calculation unit 221 generates joint velocity time series data HVT from the joint position time series data HPT.

[0137] Next, in step S23, the joint time series data imaging unit 222 images the joint position time series data HPT to generate a left hand joint position image LJPI and a right hand joint position image RJPI. Also, the joint time series data imaging unit 222 images the joint velocity time series data HVT to generate a left hand joint velocity image LJVI and a right hand joint velocity image RJVI.

[0138] Next, in step S24, the learning data generating unit 223 generates learning data sIs.

[0139] Next, in step S25, the pre-processing statistics calculation unit 224 calculates pre-processing statistics.

[0140] Finally, in step S26, the element work estimation model generation unit 225 generates the element work estimation model M using the learning data sIs and the pre-processing statistics. Finally, the element work estimation model generation unit 225 stores the element work estimation model M in the element work estimation model storage unit 232.

[0141] ***Description of the Effects of the Embodiment*** According to this embodiment, it is possible to generate an element work estimation model M for estimating an element work being performed in a work performance time period from a partial image corresponding to the work performance time period.

[0142] Embodiment 3 In this embodiment, a learning device 300 will be described as a modified example of the learning device 200 described in embodiment 2. More specifically, in this embodiment, the learning device 300 generates a trained model that is used by the work performance time zone detection unit 123 described in embodiment 1 to detect a work performance time zone. In this embodiment, differences from the second embodiment will be mainly described. It should be noted that matters not explained below are the same as those in the second embodiment.

[0143] **Configuration Description** FIG. 15 shows an example of a functional configuration of a learning device 300 according to this embodiment. In FIG. 15, an imaging device 310 is similar to the imaging device 210 shown in FIG. 12. Moreover, the joint position time-series data acquisition unit 320 is similar to the joint position time-series data acquisition unit 220 shown in FIG. 12. Moreover, the joint velocity calculation unit 321 is similar to the joint velocity calculation unit 221 shown in FIG. 12. Moreover, the joint time-series data imaging unit 322 is similar to the joint time-series data imaging unit 222 shown in FIG. For this reason, detailed explanations of these will be omitted. Similar to the learning device 200, the learning device 300 also has the hardware configuration exemplified in FIG.

[0144] **Description of the gripping tool time series data generator 325 The gripping tool time-series data generating unit 325 performs the same processing as the gripping tool time-series data generating unit 126 described in the first embodiment.

[0145] That is, the gripping tool time-series data generating unit 325 acquires the video V or the sensor information RRS. The gripped tool time-series data generator 325 uses the video V or the sensor information RRS to identify whether the worker's hand is gripping a tool. If the worker's hand is gripping a tool, the gripped tool time-series data generator 325 identifies the type of tool the worker's hand is gripping. The gripping tool time series data generating unit 325 identifies whether or not the worker's hand is holding a tool, using a method similar to that of the gripping tool time series data generating unit 126. The gripping tool time series data generating unit 325 also identifies the type of tool held by the worker's hand, using a method similar to that of the gripping tool time series data generating unit 126.

[0146] The gripped tool time-series data generating unit 325 outputs the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 indicating the classification results to the learning data generating unit 323. Further, the gripping tool time-series data generating unit 325 stores the left hand gripping tool data LTO1 and the right hand gripping tool data RTO1 in the gripping tool information storage unit 333. The left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 generated by the gripping tool time series data generation unit 325 are similar to the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 generated by the gripping tool time series data generation unit 126.

[0147] **Explanation of the learning data generation unit 323** The learning data generating unit 323 performs the same process as the work execution time zone detecting unit 123 described in the first embodiment.

[0148] That is, the learning data generation unit 323 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data visualization unit 322. Furthermore, the learning data generation unit 323 acquires the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 from the gripped tool time-series data generation unit 325.

[0149] Then, like the task execution time period detection unit 123, the learning data generation unit 323 performs appearance status determination, tool gripping status determination, and displacement amount determination. Furthermore, similar to the work performance time zone detection unit 123, the learning data generation unit 323 divides the work engagement time into a work performance time zone and a non-work time zone based on at least one of the results of the appearance status determination, the results of the tool holding status determination, and the results of the displacement amount determination. The work time in this embodiment is the work time in the learning phase, that is, the work time in this embodiment is the time during which the worker is engaged in work in the learning phase. Moreover, the work execution time period in this embodiment is a work execution time period in the learning phase, that is, a work execution time period in this embodiment is a time period during which a worker performs any one of the elemental works in the learning phase. The non-working time period in this embodiment is a non-working time period in the learning phase, i.e., a non-working time period in this embodiment is a time period in the learning phase when the worker is not performing any element work. In this embodiment, the learning data generating unit 323 detects a plurality of task execution time periods.

[0150] Next, the learning data generation unit 323 extracts, for each task performance time period, from the left hand joint position image LJPI, partial images constituting partial videos captured in each task performance time period as extracted left hand joint position images extLJPI. Furthermore, the learning data generation unit 323 extracts, from the left hand joint velocity image LJVI, for each task performance time period, partial images constituting partial videos captured in each task performance time period as an extracted left hand joint velocity image extLJVI. Furthermore, the learning data generation unit 323 extracts, for each work performance time period, from the right hand joint position image RJPI, partial images constituting partial videos captured in each work performance time period as extracted right hand joint position images extRJPI. Furthermore, the learning data generation unit 323 extracts, from the right hand joint velocity image RJVI, for each task performance time period, partial images constituting partial videos captured in each task performance time period as an extracted right hand joint velocity image extRJVI. Then, the learning data generation unit 323 stores the multiple extracted left hand joint position images extLJPI, the multiple extracted left hand joint velocity images extLJVI, the multiple extracted right hand joint position images extRJPI, and the multiple extracted right hand joint velocity images extRJVI in the learning data storage unit 330 as learning data sIt. Furthermore, the learning data generation unit 323 stores a plurality of task execution time periods in the learning data storage unit 330 as learning data sIu.

[0151] **Explanation of pre-processing statistics calculation unit 324** The pre-processing statistics calculation unit 324 obtains the learning data sIt from the learning data storage unit 330 . Then, the pre-processing statistics calculation unit 324 calculates, from the learning data sIt, the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS as pre-processing statistics. Specifically, the preprocessing statistics calculation unit 324 calculates the average luminance value in the multiple extracted left hand joint position images extLJPI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the multiple extracted left hand joint position images extLJPI is the left hand joint position luminance average value LPM(C). In addition, the preprocessing statistics calculation unit 324 calculates the average luminance value in the multiple extracted right hand joint position images extRJPI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the multiple extracted right hand joint position images extRJPI is the right hand joint position luminance average value RPM(C). Furthermore, the preprocessing statistics calculation unit 324 calculates the standard deviation of luminance in the multiple extracted left hand joint position images extLJPI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted left hand joint position image extLJPI is the left hand joint position luminance standard deviation LPS(C). In addition, the preprocessing statistics calculation unit 324 calculates the standard deviation of luminance in the multiple extracted right hand joint position images extRJPI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted right hand joint position image extRJPI is the right hand joint position luminance standard deviation RPS(C).

[0152] Furthermore, the preprocessing statistics calculation unit 324 calculates the average brightness value in the multiple extracted left hand joint velocity images extLJVI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted left hand joint velocity image extLJVI is the left hand joint velocity brightness average value LVM(C). Furthermore, the preprocessing statistics calculation unit 324 calculates the average brightness value in the multiple extracted right hand joint velocity images extRJVI included in the learning data sIt for each coordinate axis C. The average value for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted right hand joint velocity image extRJVI is the right hand joint velocity brightness average value RVM(C). Furthermore, the preprocessing statistics calculation unit 324 calculates the standard deviation of brightness in the multiple extracted left hand joint velocity images extLJVI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted left hand joint velocity image extLJVI is the left hand joint velocity brightness standard deviation LVS(C). In addition, the preprocessing statistics calculation unit 324 calculates the standard deviation of brightness in the multiple extracted right hand joint velocity images extRJVI included in the learning data sIt for each coordinate axis C. The standard deviation for each coordinate axis C calculated by the preprocessing statistics calculation unit 324 from the extracted right hand joint velocity image extRJVI is the right hand joint velocity brightness standard deviation RVS(C).

[0153] Unless a strict notation is required, the left hand joint position luminance average value LPM(C) and the right hand joint position luminance average value RPM(C) will be collectively referred to as the joint position luminance average value PM. Similarly, the left hand joint position luminance standard deviation LPS(C) and the right hand joint position luminance standard deviation RPS(C) are collectively referred to as the joint position luminance standard deviation PS. Furthermore, the left wrist joint velocity luminance average value LVM(C) and the right wrist joint velocity luminance average value RVM(C) are collectively referred to as the joint velocity luminance average value VM. Furthermore, the left hand joint velocity luminance standard deviation LVS(C) and the right hand joint velocity luminance standard deviation RVS(C) are collectively represented as the joint velocity luminance standard deviation VS.

[0154] Note that, instead of using the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand as described above, the average luminance value and standard luminance deviation for both the left hand and the right hand may be used. In the following, an example will be described in which the average luminance value and standard luminance deviation for the left hand and the average luminance value and standard luminance deviation for the right hand are used, but the following description also applies to the case in which the average luminance value and standard luminance deviation for both the left hand and the right hand are used.

[0155] The pre-processing statistics calculation unit 324 stores the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS in the pre-processing statistics storage unit 331 .

[0156] **Explanation of the work execution time period detection model generation unit 326** The work execution time zone detection model generation unit 326 acquires the learning data sIt and the learning data sIu from the learning data storage unit 330. Specifically, the task performance time zone detection model generation unit 326 acquires a plurality of extracted left hand joint position images extLJPI, a plurality of extracted left hand joint velocity images extLJVI, a plurality of extracted right hand joint position images extRJPI, and a plurality of extracted right hand joint velocity images extRJVI as the learning data sIt. Furthermore, the work performance time period detection model generation unit 326 acquires true values ​​of a plurality of work performance time periods as the learning data sIu. In addition, the task execution time period detection model generation unit 326 acquires the joint position luminance average value PM, the joint position luminance standard deviation PS, the joint velocity luminance average value VM, and the joint velocity luminance standard deviation VS from the preprocessing statistics storage unit 331. In addition, the task execution time zone detection model generation unit 326 acquires the left hand gripped tool data LTO1 and the right hand gripped tool data RTO1 from the gripped tool information storage unit 333.

[0157] Then, the work performance time zone detection model generation unit 326 generates a trained model used by the work performance time zone detection unit 123 to detect the work performance time zone as a work performance time zone detection model DM.

[0158] Specifically, the task performance time period detection model generation unit 326 resizes the extracted left hand joint position image extLJPI included in the learning data sIt into an image having a certain width and height. Furthermore, the task performance time period detection model generation unit 326 standardizes the pixel values ​​of the resized extracted left hand joint position image extLJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS. Specifically, the task performance time period detection model generation unit 326 standardizes the pixel values ​​of the resized extracted left hand joint position image extLJPI using the left hand joint position luminance average value LPM(C) and the left hand joint position luminance standard deviation LPS(C).

[0159] Furthermore, the task execution time period detection model generation unit 326 resizes the extracted right hand joint position image extRJPI included in the learning data sIt into an image having a certain width and height. Furthermore, the task performance time period detection model generation unit 326 standardizes the pixel values ​​of the resized extracted right hand joint position image extRJPI for each coordinate axis C using the joint position luminance average value PM and the joint position luminance standard deviation PS. Specifically, the task performance time period detection model generation unit 326 standardizes the pixel values ​​of the resized extracted right hand joint position image extRJPI using the right hand joint position luminance average value RPM(C) and the right hand joint position luminance standard deviation RPS(C).

[0160] Furthermore, the task execution time period detection model generation unit 326 resizes the extracted left hand joint velocity image extLJVI included in the learning data sIt into an image with a certain width and height. Furthermore, the task performance time period detection model generation unit 326 standardizes the pixel values ​​of the resized extracted left hand joint velocity image extLJVI for each coordinate axis C using the joint velocity brightness average value VM and the joint velocity brightness standard deviation VS. Specifically, the task performance time zone detection model generation unit 326 standardizes the pixel values ​​of the resized extracted left hand joint velocity image extLJVI using the left hand joint velocity brightness average value LVM(C) and the left hand joint velocity brightness standard deviation LVS(C).

[0161] Furthermore, the task execution time period detection model generation unit 326 resizes the extracted right hand joint velocity image extRJVI included in the learning data sIt into an image with a certain width and height. Furthermore, the task performance time period detection model generation unit 326 standardizes the pixel values ​​of the resized extracted right hand joint velocity image extRJVI for each coordinate axis C using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Specifically, the task performance time zone detection model generation unit 326 standardizes the pixel values ​​of the resized extracted right hand joint velocity image extRJVI using the right hand joint velocity brightness average value RVM(C) and the right hand joint velocity brightness standard deviation RVS(C). As described above, the order of resizing and standardization may be reversed.

[0162] Next, the task performance time zone detection model generation unit 326 inputs each image after resizing and standardization to a predefined neural network. The images input to the neural network are the extracted left hand joint position image after resizing and standardization extLJPI, the extracted right hand joint position image after resizing and standardization extRJPI, the extracted left hand joint velocity image after resizing and standardization extLJVI, and the extracted left hand joint velocity image after resizing and standardization extLJVI. The neural network used here is assumed to have, for example, the functions of a convolutional neural network and a region proposal network used in Faster-RCNN and the like.

[0163] Furthermore, the work performance time zone detection model generation unit 326 calculates a loss value L. The work performance time zone detection model generation unit 326 calculates, as the loss value L, the difference between the true values ​​of multiple work performance time zones included in the learning data sIu and multiple candidate values ​​(predicted values) of the work performance time zones described below, using a loss function. As the simplest loss function, the work performance time zone detection model generation unit 326 can use a squared error function. After calculating the loss value L, the work execution time period detection model generation unit 326 updates the weights of the neural network using the backpropagation method. The work performance time zone detection model generation unit 326 repeats these processes a predetermined number of times to generate the work performance time zone detection model DM. The work performance time zone detection model generation unit 326 stores the work performance time zone detection model DM in the work performance time zone detection model storage unit 332 .

[0164] **Explanation of the learning data storage unit 330** The learning data storage unit 330 stores the learning data sIt and the learning data sIu.

[0165] **Explanation of preprocessing statistics storage unit 331** The pre-processing statistics storage unit 331 stores the pre-processing statistics (joint position luminance average value PM, joint position luminance standard deviation PS, joint velocity luminance average value VM, and joint velocity luminance standard deviation VS).

[0166] **Explanation of the work execution time period detection model storage unit 332** The work performance time zone detection model storage unit 332 stores the work performance time zone detection model DM.

[0167] **Explanation of the gripping tool information storage unit 333** The holding tool information storage unit 333 stores left hand holding tool data LTO1 and right hand holding tool data RTO1.

[0168] **Explanation of an example of the internal configuration of the work execution time period detection model generation unit 326** FIG. 16 shows an example of the internal configuration of the work execution time period detection model generation unit 326. An example of the internal configuration of the work execution time period detection model generation unit 326 will be described with reference to FIG. In FIG. 16, of the functional components of the learning device 300, only the functional components necessary for explaining an example of the internal configuration of the work execution time period detection model generation unit 326 are shown.

[0169] The work performance time zone detection model generation unit 326 has, as its internal components, a joint position image acquisition unit 3261, a joint velocity image acquisition unit 3262, a joint position feature extraction unit 3263, a joint velocity feature extraction unit 3264, an appearance status determination unit 3265, a tool holding status determination unit 3266, a work performance time zone determination unit 3268, a proposal learning unit 3269, and a regression learning unit 3270.

[0170] **Explanation of the occurrence status determination unit 3265** The appearance status determination unit 3265 performs appearance status determination in the same manner as the appearance status determination unit 1231 shown in FIG.

[0171] The appearance status determination unit 3265 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging unit 322. Then, the appearance status determination unit 3265 analyzes the left hand joint position image LJPI and / or the left hand joint velocity image LJVI to generate a left hand appearance time period set LS. Similarly, the appearance status determination unit 3265 analyzes the right hand joint position image RJPI and / or the right hand joint velocity image RJVI to generate a right hand appearance time period set RS. The left hand appearance time period set LS is a set of time periods when a left hand appears in the image, and the right hand appearance time period set RS is a set of time periods when a right hand appears in the image. The appearance status determining unit 3265 outputs the left hand appearance time period set LS and the right hand appearance time period set RS to the work implementation time period determining unit 3268. The appearance status determination unit 3265 may determine the appearance status using the joint position time series data HPT and / or the joint velocity time series data HVT. In FIG. 16, the input of the joint position time series data HPT and / or the joint velocity time series data HVT to the appearance status determination unit 3265 is omitted.

[0172] **Description of the tool grip status determination unit 3266** The tool holding state determining unit 3266 performs tool holding state determination in the same manner as the tool holding state determining unit 1232 shown in FIG.

[0173] The tool holding state determination unit 3266 acquires the left hand held tool data LTO1 and the right hand held tool data RTO1 from the tool holding information storage unit 333. Then, the tool holding status determination unit 3266 extracts, in chronological order, time periods when a tool is being held and time periods when a tool is not being held for each of the left hand and right hand based on the left hand held tool data LTO1 and the right hand held tool data RTO1. Furthermore, if the type of tool held by the worker is changing, the tool holding status determination unit 3266 treats the time period during which the tool is held as a different time period for each type of tool. The tool gripping situation determination unit 3266 outputs a left hand tool gripping situation determination result LTS indicating the extraction result for the left hand and a right hand tool gripping situation determination result RTS indicating the extraction result for the right hand to the task execution time zone determination unit 3268.

[0174] **Explanation of the work execution time zone determination unit 3268** The work execution time zone determining unit 3268 performs the same process as the work execution time zone determining unit 1234 shown in FIG.

[0175] The operation implementation time zone determination unit 3268 acquires the left hand appearance time zone set LS and the right hand appearance time zone set RS from the appearance status determination unit 3265. In addition, the task execution time zone determination unit 3268 acquires the left hand tool gripping situation determination result LTS and the right hand tool gripping situation determination result RTS from the tool gripping situation determination unit 3266. Then, the work execution time zone determining unit 3268 divides the work engagement time into a work execution time zone and a non-work time zone based on these. The work implementation time zone determination unit 3268 outputs a work implementation time zone set FS, which is a set of work implementation time zones, to the joint position image acquisition unit 3261. The work implementation time zone set FS indicates the start time and end time of each work implementation time zone.

[0176] **Explanation of joint position image acquisition unit 3261** The joint position image acquisition unit 3261 performs the same processing as the joint position image acquisition unit 1241 shown in FIG.

[0177] That is, the joint position image acquisition unit 3261 acquires the work execution time period set FS from the work execution time period determination unit 3268. Furthermore, the joint position image acquisition unit 3261 acquires a left hand joint position image LJPI, a left hand joint velocity image LJVI, a right hand joint position image RJPI, and a right hand joint velocity image RJVI from the joint time-series data imaging unit 322. Furthermore, the joint position image acquisition unit 3261 acquires preprocessing statistics from the preprocessing statistics storage unit 331. Specifically, the joint position image acquisition unit 3261 acquires, as the preprocessing statistics, a joint position luminance average value PM, a joint position luminance standard deviation PS, a joint velocity luminance average value VM, and a joint velocity luminance standard deviation VS.

[0178] Then, the joint position image acquisition unit 3261 extracts partial images constituting a partial video captured during the work performance time period from each of the left hand joint position image LJPI and the right hand joint position image RJPI for each work performance time period included in the work performance time period set FS. Then, the joint position image acquisition unit 3261 resizes each partial image to an image having a certain width and height for each work time period. Furthermore, the joint position image acquisition unit 3261 standardizes the pixel values ​​of the resized partial image extracted from the left hand joint position image LJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. Furthermore, the joint position image acquisition unit 3261 standardizes the pixel values ​​of the resized partial image extracted from the right hand joint position image RJPI using the joint position luminance average value PM and the joint position luminance standard deviation PS. As described above, the order of resizing and standardization may be reversed. The joint position image acquisition unit 3261 outputs the partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI to the joint position feature extraction unit 3263.

[0179] In addition, the joint position image acquisition unit 3261 outputs the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average value VM, and the joint velocity brightness standard deviation VS to the joint velocity image acquisition unit 3262. Here, an example is described in which the joint position image acquisition unit 3261 acquires a task execution time period set FS, a left hand joint velocity image LJVI, a right hand joint velocity image RJVI, a joint velocity brightness average value VM, and a joint velocity brightness standard deviation VS, and outputs these to the joint velocity image acquisition unit 3262. Alternatively, the joint velocity image acquisition unit 3262 may acquire the task performance time period set FS from the task performance time period determination unit 3268, acquire the left hand joint velocity image LJVI and the right hand joint velocity image RJVI from the joint time series data imaging unit 322, and acquire the joint velocity brightness average value VM and the joint velocity brightness standard deviation VS from the pre-processing statistics storage unit 331. In this case, the joint position image acquisition unit 3261 acquires only the left hand joint position image LJPI and the right hand joint position image RJPI from the joint time series data imaging unit 322, and acquires only the joint position luminance average value PM and the joint position luminance standard deviation PS from the preprocessing statistics storage unit 331.

[0180] **Explanation of the joint velocity image acquisition unit 3262** The joint velocity image acquisition section 3262 performs the same processing as the joint velocity image acquisition section 1242 shown in FIG.

[0181] That is, the joint velocity image acquisition unit 3262 acquires from the joint position image acquisition unit 3261 the task execution time period set FS, the left hand joint velocity image LJVI, the right hand joint velocity image RJVI, the joint velocity brightness average value VM, and the joint velocity brightness standard deviation VS. Then, the joint velocity image acquisition unit 3262 extracts partial images constituting a partial video captured during the work performance time period from each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI for each work performance time period included in the work performance time period set FS. Then, the joint velocity image acquisition unit 3262 resizes each partial image to an image with a fixed width and height for each task time period. Furthermore, the joint velocity image acquisition unit 3262 standardizes pixel values ​​of the resized partial image extracted from the left hand joint velocity image LJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. Furthermore, the joint velocity image acquisition unit 3262 standardizes pixel values ​​of the resized partial image extracted from the right hand joint velocity image RJVI using the joint velocity luminance average value VM and the joint velocity luminance standard deviation VS. As described above, the order of resizing and standardization may be reversed. The joint velocity image acquisition unit 3262 outputs the resized and standardized partial images of each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI to the joint velocity feature extraction unit 3264.

[0182] **Explanation of the joint position feature extraction unit 3263** The joint position feature extraction unit 32633 performs the same processing as the joint position feature extraction unit 1243 shown in FIG.

[0183] That is, the joint position feature extraction unit 3263 acquires from the joint position image acquisition unit 3261 partial images after resizing and standardization for each of the left hand joint position image LJPI and the right hand joint position image RJPI. Then, the joint position feature extraction unit 3263 inputs the partial images of each of the left hand joint position image LJPI and the right hand joint position image RJPI to a pre-trained convolutional neural network. The joint position feature extraction unit 3263 extracts a joint position feature vector, which is a feature vector, from each partial image. Here, the feature vector obtained from a partial image of the left hand joint position image LJPI is called a left hand position feature vector fLP, and the feature vector obtained from a partial image of the right hand joint position image RJPI is called a right hand position feature vector fRP. The joint position feature extraction unit 3263 outputs the left hand position feature vector fLP and the right hand position feature vector fRP to the proposal learning unit 3269.

[0184] **Explanation of the joint velocity feature extraction unit 3264** The joint velocity feature extraction unit 3264 performs the same processing as the joint velocity feature extraction unit 1244 shown in FIG.

[0185] That is, the joint velocity feature extraction unit 3264 acquires from the joint velocity image acquisition unit 3262 partial images after resizing and standardization for each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI. Then, the joint velocity feature extraction unit 3264 inputs the partial images of each of the left hand joint velocity image LJVI and the right hand joint velocity image RJVI to a pre-trained convolutional neural network. Note that the convolutional neural network used here may be the same as or different from the convolutional neural network used by the joint position feature extraction unit 3263. The joint velocity feature extraction unit 3264 extracts a joint velocity feature vector, which is a feature vector, from each partial image. Here, the feature vector obtained from a partial image of the left hand joint velocity image LJVI is called a left hand velocity feature vector fLV, and the feature vector obtained from a partial image of the right hand joint velocity image RJVI is called a right hand velocity feature vector fRV. The joint velocity feature extraction unit 3264 outputs the left hand velocity feature vector fLV and the right hand velocity feature vector fRV to the proposal learning unit 3269.

[0186] **Explanation of proposed learning section 3269** The proposal learning unit 3269 acquires the left hand position feature vector fLP and the right hand position feature vector fRP from the joint position feature extraction unit 3263. In addition, the proposal learning unit 3269 acquires the left hand velocity feature vector fLV and the right hand velocity feature vector fRV from the joint velocity feature extraction unit 3264. In addition, the proposal learning unit 3269 acquires the learning data sIt from the learning data storage unit 330.

[0187] Then, the proposal learning unit 3269 converts the left hand position feature vector fLP, the right hand position feature vector fRP, the left hand velocity feature vector fLV, and the right hand velocity feature vector fRV into a map (FW×FH×FC) format. The map obtained by the proposal learning unit 3269 converting these feature vectors into a map format is called a feature map FM. Then, the proposal learning unit 3269 performs a process of proposing candidates for the work execution time period for the feature map FM. The proposal learning unit 3269 can use a proposal region network used in Faster-RCNN, which proposes multiple candidates for object existence regions, for the process of proposing candidates for work time periods. The output of the proposal process is a candidate for work time periods. A candidate for work time periods is represented by a tuple having two elements, a start time and a duration. The start time is the start time of the candidate for work time periods. Also, the duration is the duration of the candidate for work time periods. The proposal learning unit 3269 outputs a plurality of work execution time slot candidates (a set of tuples) to the regression learning unit 3270 as proposal PROPS.

[0188] **Explanation of Regression Learning Unit 3270** The regression training unit 3270 obtains the proposed PROPS from the proposed training unit 3269 . Furthermore, the regression learning unit 3270 acquires learning data sIu from the learning data storage unit 330. The learning data sIu is the true value of each of a plurality of work execution time periods obtained by the learning data generation unit 323. The learning data sIu is also a set of tuples having two elements, a start time and a duration. The start time in the learning data sIu is the start time of the work execution time period (true value). Furthermore, the duration in the learning data sIu is the duration of the work execution time period (true value).

[0189] Next, the regression learning unit 3270 obtains a tuple of the start time and duration for each candidate work performance time slot from the proposed PROPS. Furthermore, the regression learning unit 3270 uses the information of each tuple to process the partial feature map sFM extracted from the feature map FM through a fully connected layer of a convolutional neural network, and outputs an estimation result of the start time and duration of the work performance time slot. Then, the regression learning unit 3270 calculates a probability indicating a classification of whether the output result of the regression learning unit 3270 is a work performance time slot or not, and a degree of confidence that it is a work performance time slot. The regression learning unit 3270 selects a candidate having a degree of certainty greater than a predetermined threshold from among a plurality of candidates for the work execution time period. The regression learning unit 3270 may also select a candidate having an index (e.g., Intersection of Unit) value indicating the proportion of an overlapping area with a true work execution time period greater than a predetermined threshold from among a plurality of candidates for the work execution time period. Furthermore, the regression learning unit 3270 may apply a non-maximum value suppression process based on a probability that indicates the degree of certainty.

[0190] Next, the regression learning unit 3270 calculates the difference between the work implementation time slot (predicted value) of the selected candidate and the true value of the work implementation time slot acquired as the learning data sIu. The obtained difference is used as a loss value L for training the neural network in the proposal training unit 3269 and for training the neural network in the regression training unit 3270. The regression learning unit 3270 may use an L1 loss like Faster-RCNN to calculate the loss value L. The method of calculating the loss value may be any method that can appropriately calculate the difference between the predicted value and the true value of the work execution time period. After calculating the loss value L, the regression learning unit 3270 updates the weights of the neural network using the backpropagation method. The regression learning unit 3270 repeats these processes a predetermined number of times to generate an operation execution time period detection model DM. The regression learning unit 3270 stores the work performance time zone detection model DM in the work performance time zone detection model storage unit 332 .

[0191] ***Explanation of Operation*** Next, an example of the operation of learning device 300 according to this embodiment will be described. FIG. 17 is a flowchart showing an example of the operation of the learning device 300.

[0192] First, in step S31, the joint position time-series data acquisition unit 320 generates joint position time-series data HPT from the video V from the imaging device 310.

[0193] Next, in step S32, the joint velocity calculation unit 321 generates joint velocity time series data HVT from the joint position time series data HPT.

[0194] Next, in step S33, the joint time series data imaging unit 322 images the joint position time series data HPT to generate a left hand joint position image LJPI and a right hand joint position image RJPI. The joint time series data imaging unit 322 also images the joint velocity time series data HVT to generate a left hand joint velocity image LJVI and a right hand joint velocity image RJVI.

[0195] Next, in step S34, the learning data generating unit 323 generates learning data sIt and learning data sIu.

[0196] Next, in step S35, the pre-processing statistics calculation unit 324 calculates pre-processing statistics.

[0197] Finally, in step S36, the work performance time zone detection model generation unit 326 generates the work performance time zone detection model DM using the learning data sIt, the learning data sIu, and the pre-processing statistics. Finally, the work performance time zone detection model generation unit 326 stores the work performance time zone detection model DM in the element work estimation model storage unit 232.

[0198] ***Description of the Effects of the Embodiment*** According to this embodiment, the work performance time zone detection model DM can be generated based on the learning data sIt and the learning data sIu. This eliminates the need to perform rule-based detection of the work performance time zone. In other words, it is no longer necessary to determine a threshold value for detecting the work performance time zone. As a result, the burden on the user of the estimation device 100 can be reduced.

[0199] Embodiment 4 FIG. 18 shows an example of a functional configuration of an estimation device 400 according to this embodiment. In FIG. 18, an work performance time zone detection model storage unit 433 stores the work performance time zone detection model DM shown in the third embodiment. Furthermore, the work performance time zone detection unit 423 acquires the work performance time zone detection model DM from the work performance time zone detection model storage unit 433, and detects the work performance time zone using the work performance time zone detection model DM. That is, the work performance time zone detection unit 423 inputs the left hand joint position image LJPI, left hand joint velocity image LJVI, right hand joint position image RJPI, right hand joint velocity image RJVI, left hand held tool data LTO1, and right hand held tool data RTO1 into the work performance time zone detection model DM to detect the work performance time zone. The components other than the work execution time zone detection unit 423 and the work execution time zone detection model storage unit 433 are similar to the components with the same names shown in Fig. 7. Therefore, the description of these components will be omitted. Similarly to the estimation device 100, the estimation device 400 also has the hardware configuration exemplified in FIG.

[0200] ***Description of the Effects of the Embodiment*** As in the third embodiment, it is no longer necessary to determine a threshold value for detecting an operation execution time period, so that the burden on the user of the estimation device 400 can be reduced. Furthermore, by using the task execution time zone detection model DM, the task execution time zone can be accurately detected even for tasks in which element tasks are combined in a complex manner.

[0201] Embodiment 5. 19 shows an example of a functional configuration of a learning device 500 according to this embodiment. Learning device 500 is a combination of learning device 200 described in the second embodiment and learning device 300 described in the third embodiment. The learning data generating unit 523 generates the learning data sIs, the learning data sIt, and the learning data sIu. The method of generating the learning data sIs, the learning data sIt, and the learning data sIu is the same as that described in the second and third embodiments. Moreover, the preprocessing statistics calculation unit 524 calculates preprocessing statistics of each of the learning data sIs and the learning data sIt. The method of calculating the preprocessing statistics is the same as that described in the second and third embodiments. The components other than the learning data generation unit 523 and the pre-processing statistics calculation unit 524 are similar to the components of the same names shown in Fig. 12 and Fig. 15. Therefore, the description of these components will be omitted. Similarly to the learning devices 200 and 300, the learning device 500 also has the hardware configuration exemplified in FIG.

[0202] The learning for generating the work execution time period detection model DM and the learning for generating the element work estimation model M may be performed independently of each other, or may be performed simultaneously as multitask learning. When learning for generating the task performance time zone detection model DM and learning for generating the element task estimation model M are performed simultaneously, part of the network structure and / or weights may be shared between the two learnings.

[0203] Furthermore, for the order of multiple element tasks, which is the detection and classification result as the estimation result, a loss value L for the order may be calculated by comparing the estimation result with the true value (the true order of the element tasks), and the loss value L may be used for training each model. The loss value L for the order may be calculated using, for example, the edit distance or the difference between graphs. It is also desirable to change the calculation method of the loss value L for the sequence depending on the type of work. For example, when calculating the loss value L for an assembly work, it is possible to impose a large loss value L on the occurrence of an impossible sequence of elemental work in carrying out the assembly work.

[0204] ***Description of the Effects of the Embodiment*** According to this embodiment, the work execution time period detection model DM and the element work estimation model M can be generated efficiently.

[0205] Although the first to fifth embodiments have been described above, two or more of these embodiments may be combined for implementation. Alternatively, one of these embodiments may be partially implemented. Alternatively, two or more of these embodiments may be partially combined and implemented. Furthermore, the configurations and procedures described in these embodiments may be modified as necessary.

[0206] ***Additional hardware configuration information*** Finally, a supplementary explanation of the hardware configurations of the estimation device 100, the learning device 200, the learning device 300, the estimation device 400, and the learning device 500 will be given. Below, a supplementary explanation will be given of the hardware configurations of the estimation device 100 and the learning device 200. The following explanation of the estimation device 100 also applies to the estimation device 400. Similarly, the following explanation of the learning device 200 also applies to the learning device 300 and the learning device 500.

[0207] A processor 801 shown in FIG. 22 is an IC (Integrated Circuit) that performs processing. The processor 801 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or the like. The main memory device 802 shown in FIG. 22 is a RAM (Random Access Memory). The auxiliary storage device 803 shown in FIG. 22 is a ROM (Read Only Memory), a flash memory, a HDD (Hard Disk Drive), or the like. The communication device 804 shown in FIG. 22 is an electronic circuit that performs data communication processing. The communication device 804 is, for example, a communication chip or a NIC (Network Interface Card).

[0208] The auxiliary storage device 803 also stores an OS (Operating System). At least a portion of the OS is executed by the processor 801 . The processor 801 executes a program that realizes each functional component of the estimation device 100 while executing at least a part of the OS. The processor 801 executes the OS, which performs task management, memory management, file management, communication control, and the like. In addition, at least one of information, data, signal values, and variable values ​​indicating the results of processing of each functional component of the estimation device 100 is stored in at least one of the main memory device 802, the auxiliary memory device 803, and a register and cache memory within the processor 801. Furthermore, the programs for realizing the respective functional components of the estimation device 100 may be stored in portable recording media such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, a DVD, etc. Then, the portable recording media in which the programs for realizing the respective functional components of the estimation device 100 are stored may be distributed.

[0209] Furthermore, at least any "part" of the functional components of the estimation device 100 may be read as a "circuit" or a "step" or a "procedure" or a "process" or a "circuitry". Furthermore, the estimation device 100 may be realized by a processing circuit. The processing circuit is, for example, a logic integrated circuit (IC), a gate array (GA), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). In this case, the functional components of the estimation device 100 are each implemented as part of a processing circuit.

[0210] The processor 901 shown in FIG. 23 is an IC that performs processing. The processor 901 is a CPU, a DSP, or the like. The main memory device 902 shown in FIG. 23 is a RAM. The auxiliary storage device 903 shown in FIG. 23 is a ROM, a flash memory, a HDD, or the like. The communication device 904 shown in FIG. 23 is an electronic circuit that executes data communication processing. The communication device 904 is, for example, a communication chip or a NIC.

[0211] The auxiliary storage device 903 also stores the OS. At least a part of the OS is executed by the processor 901 . The processor 901 executes at least a part of the OS and executes programs that realize each functional component of the learning device 200. The processor 901 executes the OS, which performs task management, memory management, file management, communication control, and the like. In addition, at least one of information, data, signal values, and variable values ​​indicating the results of processing of each functional component of the learning device 200 is stored in at least one of the main memory device 902, the auxiliary memory device 903, a register within the processor 901, and a cache memory. Furthermore, the programs that realize the functional components of learning device 200 may be stored on portable recording media such as magnetic disks, flexible disks, optical disks, compact disks, Blu-ray (registered trademark) disks, DVDs, etc. Portable recording media on which the programs that realize the functional components of learning device 200 are stored may be distributed.

[0212] Furthermore, at least any "part" of the functional components of the learning device 200 may be read as a "circuit" or a "step" or a "procedure" or a "process" or a "circuitry". Furthermore, the learning device 200 may be realized by a processing circuit, such as a logic IC, a GA, an ASIC, or an FPGA. In this case, the functional components of the learning device 200 are each realized as part of a processing circuit.

[0213] In this specification, the higher-level concept of a processor and a processing circuit is called "processing circuitry." That is, a processor and a processing circuit are each specific examples of "processing circuitry." [Explanation of symbols]

[0214] 11 time division unit, 12 estimation unit, 100 estimation device, 110 imaging device, 120 joint position time series data acquisition unit, 121 joint velocity calculation unit, 122 joint time series data imaging unit, 123 work execution time zone detection unit, 1231 appearance status determination unit, 1232 tool gripping status determination unit, 1233 displacement amount determination unit, 1234 work execution time zone determination unit, 124 element work estimation unit, 1241 joint position image acquisition unit, 1242 joint velocity image acquisition unit, 1243 joint position feature extraction unit, 1244 joint velocity feature extraction unit, 1245 joint image feature classification unit, 125 estimation result processing unit, 126 gripping tool time series data generation unit, 130 element work estimation model storage unit, 131 pre-processing statistics storage unit, 132 estimation result storage unit, 200 learning device, 210 imaging device, 220 Joint position time series data acquisition unit, 221 joint velocity calculation unit, 222 joint time series data imaging unit, 223 learning data generation unit, 224 pre-processing statistics calculation unit, 225 element work estimation model generation unit, 2251 joint position image acquisition unit, 2252 joint velocity image acquisition unit, 2253 joint position feature extraction unit, 2254 joint velocity feature extraction unit, 2255 joint image feature classification learning unit, 230 learning data storage unit, 231 pre-processing statistics storage unit, 232 element work estimation model storage unit, 300 learning device, 310 imaging device, 320 joint position time series data acquisition unit, 321 joint velocity calculation unit, 322 joint time series data imaging unit, 323 learning data generation unit, 324 pre-processing statistics calculation unit, 325 gripping tool time series data generation unit, 326 work execution time zone detection model generation unit, 3261 Joint position image acquisition unit, 3262 joint velocity image acquisition unit, 3263 joint position feature extraction unit, 3264 joint velocity feature extraction unit, 3265 appearance status determination unit, 3266 tool gripping status determination unit, 3268 task execution time zone determination unit, 3269 proposal learning unit, 3270 regression learning unit, 330 learning data storage unit, 331 pre-processing statistics storage unit, 332 task execution time zone detection model storage unit, 333 gripping tool information storage unit, 400 estimation device, 410 imaging device, 420 joint position time series data acquisition unit, 421 joint velocity calculation unit, 422 joint time series data imaging unit, 423 task execution time zone detection unit, 424 element work estimation unit, 425 estimation result processing unit, 426 gripping tool time series data generation unit, 430Element work estimation model storage unit, 431 estimation result storage unit, 432 pre-processing statistics storage unit, 433 work execution time zone detection model storage unit, 500 learning device, 510 imaging device, 520 joint position time series data acquisition unit, 521 joint velocity calculation unit, 522 joint time series data imaging unit, 523 learning data generation unit, 524 pre-processing statistics calculation unit, 525 work execution time zone detection model generation unit, 526 grip tool time series data generation unit, 527 element work estimation model generation unit, 530 learning data storage unit, 531 pre-processing statistics storage unit, 532 work execution time zone detection model storage unit, 533 grip tool information storage unit, 534 element work estimation model storage unit, 801 processor, 802 main storage unit, 803 auxiliary storage unit, 804 communication device, 901 processor, 902 main storage unit, 903 auxiliary storage unit, 904 Communications equipment.

Claims

1. A displacement amount determination result for determining the displacement amount of the hand of a worker at the working time, which is the time when the worker is engaged in work including a plurality of element operations with non-uniform attributes for each attribute; a tool gripping state determination result for determining the gripping state of the tool in the hand of the worker at the working time; and an appearance state determination result for determining the appearance state of the hand of the worker in the video imaged during the working time. Using at least any one of these, the working time is divided into a working execution time zone, which is a time zone when the worker is performing any one of the plurality of element operations, and a non-working time zone, which is a time zone when the worker is not performing any of the plurality of element operations. A time division unit for dividing; A plurality of partial images constituting a partial video, which is a part of the video imaged during the working execution time zone, are obtained, and the plurality of partial images are standardized using the average value and standard deviation of the luminance of the learned hand image used for learning when generating the learned model, and resized to a specified size. The plurality of processed partial images, which are the plurality of partial images after standardization and resizing, are input into the learned model to extract the feature vectors of the plurality of processed partial images. An estimation device having an estimation unit that inputs the feature vectors of the plurality of processed partial images into the learned model to estimate the element operations performed by the worker during the working execution time zone.

2. The estimation unit As the plurality of partial images, a hand position partial image, which is a partial image of the position of the hand of the worker, and a hand speed partial image, which is a partial image of the speed of the hand of the worker, are obtained. The hand position partial image is standardized using the average value and standard deviation of the luminance of the learned image of the hand position used for learning when generating the learned model, and resized to the specified size. The hand speed partial image is standardized using the average value and standard deviation of the luminance of the learned image of the hand speed used for learning when generating the learned model, and resized to the specified size. The processed hand position partial image, which is the hand position partial image after standardization and resizing, is input into the learned model to extract the feature vector of the processed hand position partial image. Input the processed hand speed partial image, which is the hand speed partial image after standardization and resizing, into the learned model to extract the feature vector of the processed hand speed partial image. The estimation device according to claim 1, wherein the feature vector of the processed hand position partial image and the feature vector of the processed hand speed partial image are combined, and the combined feature vector is input into the learned model to estimate the elemental work performed by the worker during the work execution time period.

3. The time division unit When using the determination result of the displacement amount determination, the time period during which the displacement amount of the worker's hand is less than the threshold value among the work engagement times is designated as the work execution time period in the estimation device according to claim 1.

4. The time division unit When using the determination result of the tool gripping state determination, the time period during which the worker's hand is gripping the tool among the work engagement times is designated as the work execution time period in the estimation device according to claim 1.

5. The time division unit When the type of the tool gripped by the worker's hand changes over time, each time period during which the worker's hand is gripping different types of tools is designated as a different work execution time period in the estimation device according to claim 4.

6. The time division unit When using the determination result of the appearance situation determination, the time period during which the worker's hand appears in the video among the work engagement times is designated as the work execution time period in the estimation device according to claim 1.

7. The time division unit Apply at least any one of the determination result of the displacement amount determination, the determination result of the tool gripping state determination, and the determination result of the appearance situation determination to the learned model to divide the work engagement time into the work execution time period and the non-work time period in the estimation device according to claim 1.

8. Generate candidates for the work execution time period, A learning device that generates the learned model of claim 1 by learning the difference between the true value of the work execution time period obtained by performing at least any one of the process corresponding to the displacement amount determination, the process corresponding to the tool gripping state determination, and the process corresponding to the appearance situation determination, and the candidate for the work execution time period.

9. Calculate the average value and standard deviation of the luminance of the learning image, A learning device that generates the learned model of claim 1 by learning using the calculated average value and standard deviation.

10. Using at least any one of the determination result of displacement amount determination for determining the displacement amount of the hand of the worker during the working time, which is the time when the worker is engaged in work including a plurality of element operations with non-uniform attributes for each element operation, the determination result of tool gripping state determination for determining the gripping state of the tool in the hand of the worker during the working time, and the determination result of appearance state determination for determining the appearance state of the hand of the worker in the video captured during the working time, a computer divides the working time into a work execution time period, which is a time period when the worker is performing any one of the plurality of element operations, and a non-working time period, which is a time period when the worker is not performing any of the plurality of element operations. The computer acquires a plurality of partial images constituting a partial video, which is a part of the video captured during the work execution time period, and performs normalization using the average value and standard deviation of the luminance of the learned hand image used for learning during the generation of the learned model and resizing to a specified size on the plurality of partial images. The computer inputs the plurality of processed partial images, which are the plurality of partial images after normalization and resizing, into the learned model to extract the feature vectors of the plurality of processed partial images. An estimation method for estimating the element operation that the worker is performing during the work execution time period by inputting the feature vectors of the plurality of processed partial images into the learned model.

11. A time division process for dividing the working time into a work execution time period, which is a time period when the worker is performing any one of the plurality of element operations, and a non-working time period, which is a time period when the worker is not performing any of the plurality of element operations, using at least any one of the determination result of displacement amount determination for determining the displacement amount of the hand of the worker during the working time, which is the time when the worker is engaged in work including a plurality of element operations with non-uniform attributes for each element operation, the determination result of tool gripping state determination for determining the gripping state of the tool in the hand of the worker during the working time, and the determination result of appearance state determination for determining the appearance state of the hand of the worker in the video captured during the working time. Obtain a plurality of partial images that constitute a partial video, which is a portion of the video captured during the work execution time period, and perform normalization on the plurality of partial images using the average value and standard deviation of the luminance of the learned hand images used in the learning during the generation of the learned model, and resizing to a specified size. Input the plurality of processed partial images, which are the plurality of partial images after normalization and resizing, into the learned model to extract the feature vectors of the plurality of processed partial images. An estimation program that causes a computer to execute an estimation process of inputting the feature vectors of the plurality of processed partial images into the learned model to estimate the elemental work being performed by the worker during the work execution time period.

Images