Operation Analysis Device

Abstract

The present invention recognizes an object from an image to classify work with a small computation amount.　Provided is a work analysis device for analyzing work of a worker, the work analysis device comprising: a joint position estimation unit that estimates joint position information relating to the worker from video data including the work of the worker; a motion estimation unit that estimates motion information relating to the worker on the basis of the joint position information estimated by the joint position estimation unit; an image extraction unit that extracts a range of the video data relating to an object relevant to the motion information from the video data on the basis of the motion information estimated by the motion estimation unit; an object recognition unit that recognizes the object in the range of the video data extracted by the image extraction unit; and a work identification unit that identifies the work of the worker on the basis of the object recognized by the object recognition unit.

Description

[Technical Field]

[0001] The present invention relates to a work analysis device. [Background technology]

[0002] In factories, operational data on machine tools and other equipment can be obtained, but data on worker work cannot be obtained. Therefore, in order to improve work, consider introducing robots, and realize digital twins of factories, it is necessary to visualize the work of workers, and technology that can automatically recognize what workers were doing from video footage of their work is important. In this regard, there is known a technology that performs machine learning using learning data consisting of input data of images of workers performing tasks and label data of the tasks performed by the workers in the images, generates a trained model for identifying tasks from images, and uses the trained model to identify which task is being performed in the image being analyzed (see, for example, Patent Document 1). Also, there is known a technology for identifying the position of a worker's hand from image data with depth captured by a depth sensor, and for identifying the position of an object from image data captured by a digital camera, and for identifying the details of the action performed by the worker during work. For example, see Patent Document 2. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-67981 [Patent Document 2] International Publication No. 2017 / 222070 Summary of the Invention [Problem to be solved by the invention]

[0004] However, classification models such as the trained model in Patent Document 1 have the problem of being complex and having low interpretability. Furthermore, in order to detect the tools (objects) used in an image for work classification as in Patent Document 2, a large amount of calculation is required because the entire image must be scanned.

[0005] Therefore, it is desirable to classify tasks by recognizing objects from images with a small amount of calculation. [Means for solving the problem]

[0006] One aspect of the work analysis device disclosed herein is a work analysis device that analyzes work of a worker, and includes: a joint position estimation unit that estimates joint position information of the worker from video data including the work of the worker; a movement estimation unit that estimates movement information of the worker based on the joint position information estimated by the joint position estimation unit; an image cropping unit that crops out a range of video data relating to an object associated with the movement information from the video data based on the movement information estimated by the movement estimation unit; an object recognition unit that recognizes the object within the range of the video data cropped by the image cropping unit; and a work identification unit that identifies the work of the worker based on the object recognized by the object recognition unit.

[0007] an object area entry / exit detection unit that detects whether an image area including joint positions of the worker has entered or exited an image area including the object detected by the object detection unit based on the detection result of the object area entry / exit detection unit; an image cropping unit that crops out from the video data a range of video data related to the object detected by the object detection unit based on the detection result of the object area entry / exit detection unit; an object recognition unit that performs object recognition within the range of video data cropped by the image cropping unit; an object detection activation unit that, if the object recognition unit cannot recognize the object within the range of the video data, causes the object detection unit to periodically detect the object; and a work estimation unit that identifies a task based on a change in the coordinates of the object detected by the object detection unit in the video data. [Effects of the Invention]

[0008] According to one aspect, it is possible to recognize objects from images and classify tasks with a small amount of calculation. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a functional block diagram showing an example of the functional configuration of an operation analysis system according to a first embodiment. [Figure 2A] FIG. 10 is a diagram showing an example of a range on video data according to motion information of a worker and a tool (object). [Figure 2B] FIG. 10 is a diagram showing an example of a range on video data according to motion information of a worker and a tool (object). [Figure 3] FIG. 10 is a diagram illustrating an example of a working table. [Figure 4A] FIG. 1 is a diagram showing an example of the shape of a hand gripping a driver. [Figure 4B] FIG. 4B is a diagram showing an example of a hand shape similar to that of FIG. 4A for gripping a caliper. [Figure 5A] 2C is a diagram showing an example of video data cut out when the shape of the worker's hands is that of a screwdriver's hand from the video data shown in FIG. 2B. FIG. [Figure 5B] 2C is a diagram showing an example of video data cut out from the video data shown in FIG. 2B when the shape of the worker's hand is that of a hand using a caliper. FIG. [Figure 6] 10 is a flowchart illustrating an analysis process of the work analysis device. [Figure 7] FIG. 10 is a functional block diagram showing an example of the functional configuration of an operation analysis system according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of video data including work performed by a worker. [Figure 9] FIG. 10 is a diagram illustrating an example of video data including work performed by a worker. [Figure 10] FIG. 10 is a diagram illustrating an example of video data including work performed by a worker. [Figure 11] FIG. 10 is a diagram illustrating an example of video data including work performed by a worker. [Figure 12] 10 is a flowchart illustrating an analysis process of the work analysis device. DETAILED DESCRIPTION OF THE INVENTION

[0010] A first embodiment and a second embodiment of the work analysis device will be described in detail with reference to the drawings. Here, each embodiment has in common a configuration in which the work of a worker is identified from an image of the worker and an object (tool) captured by a camera. However, in identifying the work of a worker in the first embodiment, joint position information of the worker is estimated from video data including the work of the worker, motion information of the worker is estimated based on the estimated joint position information of the worker, a range of video data relating to an object associated with the motion information is extracted from the video data based on the estimated motion information of the worker, the object is recognized within the extracted range of video data, and the work of the worker is identified from the recognized object. In contrast, the second embodiment differs from the first embodiment in that it detects an object from video data including the work of the worker, estimates joint position information of the worker, detects whether the joint position of the worker has entered or exited an image area including the detected object based on the estimated joint position information of the worker, extracts a range of video data relating to the detected object from the video data based on the detection result, performs object recognition within the extracted range of video data, and periodically performs object detection for the object if the object cannot be recognized within the range of the video data, thereby determining the work of the worker based on changes in the coordinates of the object. In the following, the first embodiment will be described in detail first, and then the second embodiment will be described, focusing on the differences from the first embodiment.

[0011] First Embodiment FIG. 1 is a functional block diagram showing an example of the functional configuration of the work analysis system according to the first embodiment. As shown in FIG. 1, the work analysis system 100 includes a work analysis device 1 and a camera 2.

[0012] The work analysis device 1 and camera 2 may be connected to each other via a network (not shown) such as a LAN (Local Area Network) or the Internet. In this case, the work analysis device 1 and camera 2 are equipped with a communication unit (not shown) for communicating with each other via such a connection. The work analysis device 1 and camera 2 may also be directly connected to each other via a connection interface (not shown) via a wired or wireless connection. Furthermore, in FIG. 1, the work analysis device 1 is connected to one camera 2, but it may be connected to two or more cameras 2.

[0013] The camera 2 is a digital camera or the like, and captures two-dimensional frame images of workers, tools, and other objects (not shown) projected onto a plane perpendicular to the optical axis of the camera 2 at a predetermined frame rate (e.g., 30 fps). The camera 2 outputs the captured frame images as video data to the work analysis device 1. Note that the video data captured by the camera 2 may be visible light images such as RGB color images, grayscale images, and depth images.

[0014] <Work analysis device 1> The task analysis device 1 is a computer known to those skilled in the art, and as shown in Fig. 1, includes a control unit 10 and a storage unit 20. The control unit 10 also includes a joint position estimation unit 101, a motion estimation unit 102, an image clipping unit 103, an object recognition unit 104, and a task identification unit 105. The task identification unit 105 also includes a task estimation unit 1051.

[0015] The storage unit 20 is a storage device such as a ROM (Read Only Memory) or an HDD (Hard Disk Drive). The storage unit 20 stores an operating system and application programs executed by the control unit 10 (described later). The storage unit 20 also includes a video data storage unit 201, an action storage unit 202, an object positional relationship storage unit 203, and a work storage unit 204.

[0016] The video data storage unit 201 stores video data of workers and objects such as tools captured by the camera 2.

[0017] The movement storage unit 202 stores a rule base or a trained model that outputs movement information of a worker corresponding to joint position information of the worker estimated by the movement estimation unit 102, which will be described later. Specifically, for example, joint position information including joint positions of the worker's hands, etc., in video data of a worker performing each task to be identified (e.g., "measuring with a caliper" or "driving a screw") captured by the camera 2 is used as input data, and a trained model such as a neural network that is generated in advance by known machine learning using training data in which the task is used as label data may be stored in the movement storage unit 202. Alternatively, the movement storage unit 202 may store a rule base that associates the task with joint position information of the worker in video data of the worker performing each task to be identified captured by the camera 2, based on a known method.

[0018] The object positional relationship storage unit 203 stores in advance, based on the motion information of the worker estimated by the motion estimation unit 102 described later, the range of the video data that includes the tool (object) related to the motion information. 2A and 2B are diagrams showing examples of ranges on video data corresponding to the worker's motion information and the tool (object). Fig. 2A shows an image in which the worker is measuring with a caliper as the motion information. Fig. 2B shows an image in which the worker is turning a screw with a screwdriver as the motion information. As shown in FIG. 2A , when a worker is performing measurements with a caliper, the object position relation storage unit 203 pre-stores, as a range on the video data where the caliper (object) exists, relative position coordinates in an image coordinate system of a rectangle long in the horizontal direction indicated by a dashed line, for example, based on the joint position of the worker's hand (dashed rectangle) indicated by joint position information estimated by the joint position estimation unit 101 described later. Furthermore, as shown in FIG. 2B, when a worker is turning a screw, the relative position coordinates in the image coordinate system of a rectangle elongated in the vertical direction indicated by a dot-dash line are stored in advance in the object position relation storage unit 203 as the range on the video data where a screwdriver (object) exists, based on the joint position (dashed rectangle) of the worker's hand indicated by the joint position information estimated by the joint position estimation unit 101 described later.

[0019] The work storage unit 204 stores a work table that associates tools (objects) recognized by the object recognition unit 104 (described later) with the work of the corresponding worker. FIG. 3 is a diagram illustrating an example of the working table. As shown in FIG. 3, the work table has storage areas for "objects" and "work." The storage area for "object" in the work table stores names of tools such as "screwdriver" and "caliper." In the storage area for "task" in the task table, tasks such as "screw driving" and "measure with calipers" are stored. The storage areas for "objects" and "tasks" in the work table may be configured so that users such as workers can register them in advance using an input device such as a keyboard or touch panel included in the work analysis device 1.

[0020] The control unit 10 includes a CPU, a ROM, a RAM (Random Access Memory), a CMOS memory, and the like, which are configured to be able to communicate with each other via a bus, and are well known to those skilled in the art. The CPU is a processor that provides overall control of the work analysis device 1. The CPU reads system programs and application programs stored in ROM via the bus and controls the entire work analysis device 1 in accordance with the system programs and application programs. As a result, as shown in FIG. 1 , the control unit 10 is configured to implement the functions of a joint position estimation unit 101, a movement estimation unit 102, an image clipping unit 103, an object recognition unit 104, and a work identification unit 105. The work identification unit 105 is also configured to implement the function of a work estimation unit 1051. The RAM stores various data such as temporary calculation data and display data. The CMOS memory is backed up by a battery (not shown) and is configured as non-volatile memory that retains its stored state even when the work analysis device 1 is powered off.

[0021] The joint position estimation unit 101 estimates the joint position information of the worker from video data including the worker's work. Specifically, the joint position estimation unit 101 uses a known method (e.g., Kosuke Kanno, Kenta Oku, and Kyoji Kawagoe, "Motion Detection and Classification Method from Multidimensional Time Series Data," DEIM Forum 2016 G4-5, or Shohei Uezono and Satoshi Ono, "Feature Extraction of Multimodal Sequence Data Using LSTM Autoencoder," Materials from the Japanese Society for Artificial Intelligence Study Group, SIG-KBS-B802-01, 2018) to estimate time series data of coordinates and angles (hand shape) of joints of the worker's hands, etc., as joint position information from video data with time information added thereto stored in the video data storage unit 201. In the following, a case will be described in which the joint position estimation unit 101 estimates the joint positions of the worker's hands as joint position information. However, the joint position estimation unit 101 can also estimate the joint positions of parts of the worker other than the hands in the same way as the joint positions of the hands.

[0022] The movement estimation unit 102 estimates movement information of the worker based on the joint position information estimated by the joint position estimation unit 101. In the following, a case will be described in which the movement estimation unit 102 estimates movement information of "measuring with calipers" shown in Fig. 2A and "driving a screw" shown in Fig. 2B as worker movements. However, the movement estimation unit 102 also estimates movement information other than "measuring with calipers" and "driving a screw" in the same way as for "measuring with calipers" and "driving a screw." Specifically, the movement estimation unit 102 inputs, for example, joint position information indicating the hand shape estimated by the joint position estimation unit 101 as input data into a trained model stored in the movement storage unit 202, and estimates the movement of the worker in the video data (i.e., "measuring with calipers" or "driving a screw"). Alternatively, the movement estimation unit 102 may estimate the movement of the worker in the video data based on the joint position information indicating the hand shape estimated by the joint position estimation unit 101 and a rule base stored in the movement storage unit 202. Furthermore, the movement estimation unit 102 may calculate, together with the estimated movement information of the worker, a probability indicating the likelihood of the hand shape (hand joint position) performing the movement indicated by the movement information. 4A and 4B, when the hand shape estimated by the joint position estimation unit 101 is vague and resembles the joint positions of gripping two or more different objects (tools), the movement estimation unit 102 may estimate a plurality of movements as movement information. Fig. 4A is a diagram showing an example of a hand shape gripping a screwdriver. Fig. 4B is a diagram showing an example of a hand shape gripping a vernier caliper, similar to the hand shape in Fig. 4A.

[0023] The image cutout unit 103 cuts out a range of video data relating to an object (tool) associated with the motion information from the video data based on the motion information estimated by the motion estimation unit 102. Specifically, the image cropping unit 103 acquires, for example, from the object positional relation storage unit 203, relative position coordinates in the image coordinate system, which is the range on the video data to be cropped that corresponds to the motion information estimated by the motion estimation unit 102. As shown in FIG. 2A or 2B , the image cropping unit 103 crops out the video data of the rectangular range indicated by the dashed line based on the relative position coordinates acquired with reference to the joint position of the worker's hand (dashed rectangle). In addition, when the movement information estimated by the movement estimation unit 102 includes multiple movements, the image cropping unit 103 acquires relative position coordinates in the image coordinate system corresponding to each of the multiple movements indicated by the movement information, and crops out video data of a rectangular range based on the relative position coordinates of each movement acquired with reference to the joint positions of the worker's hand. 5A and 5B are diagrams showing an example of extracted video data when the action information includes a plurality of actions. Fig. 5A is a diagram showing an example of video data extracted from the video data shown in Fig. 2B when the shape of the worker's hand is that of a hand used for a screwdriver. Fig. 5B is a diagram showing an example of video data extracted from the video data shown in Fig. 2B when the shape of the worker's hand is that of a hand used for a vernier caliper.

[0024] The object recognition unit 104 recognizes an object (tool) within the range of the video data cut out by the image cutout unit 103. Specifically, the object recognition unit 104 extracts image features such as edge amounts from the extracted video data using, for example, a known method. The object recognition unit 104 performs a matching process between the extracted image features and image features for each tool (object) pre-stored in the storage unit 20, and recognizes the tool (object) in the extracted video data. The object recognition unit 104 may also calculate a probability indicating the likelihood of the recognized tool (object). For example, when the motion information estimated by the motion estimation unit 102 includes multiple motions, the object recognition unit 104 may recognize a driver (object) from the range of the clipped video data in Fig. 5A and calculate the probability of the driver (object) as 90%. Also, the object recognition unit 104 may not be able to recognize a caliper (tool) from the range of the clipped video data in Fig. 5B and calculate the probability of the caliper (object) as 3%.

[0025] The task identification unit 105 identifies the task of the worker based on the object (tool) recognized by the object recognition unit 104. Specifically, the task identification unit 105 identifies the task of the worker based on, for example, the tool (object) recognized by the object recognition unit 104 and the task table stored in the task memory unit 204. The task identification unit 105 may display the identified task on a display device (not shown), such as a liquid crystal display, included in the work analysis device 1. Furthermore, if the tool (object) recognized by the object recognition unit 104 is not registered in the work table stored in the work memory unit 204, the work identification unit 105 may display a message such as "The work could not be identified" on the display device (not shown) of the work analysis device 1.

[0026] When the movement information estimated by the movement estimation unit 102 includes multiple movements, the movement estimation unit 1051 estimates the most probable movement based on the probability of the hand shape (hand joint position) performing each of the multiple movements estimated by the movement estimation unit 102 and the probability of the object recognized for each range of the multiple video data extracted by the object recognition unit 104. For example, in the video data shown in FIG. 5A, if the probability of the hand shape (hand joint position) performing the action of "screwdriving" estimated by the action estimation unit 102 is 60% and the probability of the "screwdriver" recognized by the object recognition unit 104 is 90%, the task estimation unit 1051 calculates the probability of the task of "screwdriving" to be 0.5 (= 0.6 × 0.9). Also, in the video data shown in FIG. 5B, if the probability of the hand shape (hand joint position) performing the action of "measuring with calipers" estimated by the action estimation unit 102 is 40% and the probability of the "calipers" recognized by the object recognition unit 104 is 3%, the task estimation unit 1051 calculates the probability of the task of "measuring with calipers" to be 0.01 (= 0.4 × 0.03). Then, the task estimation unit 1051 identifies "screwdriving," which has the highest probability of 0.5, as the task of the worker.

[0027] <Analysis process of work analysis device 1> Next, the operation of the analysis process of the work analysis device 1 according to the first embodiment will be described. 6 is a flowchart illustrating the analysis process of the work analysis device 1. The flow shown here is repeatedly executed while video data is being input from the camera 2.

[0028] In step S1, the joint position estimation unit 101 estimates the joint position information of the worker's hand from video data including the worker's work.

[0029] In step S2, the movement estimating unit 102 estimates movement information of the worker based on the joint position information estimated in step S1.

[0030] In step S3, the image cropping unit 103 crops out a range of video data relating to an object (tool) related to the action included in the action information estimated in step S2. Note that, when the action information estimated in step S2 includes multiple actions, the image cropping unit 103 crops out a range of video data relating to an object (tool) related to each action.

[0031] In step S4, the object recognition unit 104 recognizes an object (tool) within the range of the video data extracted in step S3. If there are multiple pieces of video data extracted in step S3, the object recognition unit 104 recognizes an object (tool) within the range of each of the multiple pieces of video data.

[0032] In step S5, the task identification unit 105 identifies the task of the worker based on the tool (object) recognized in step S4 and the task table stored in the task memory unit 204. If multiple tasks are estimated by the task estimation unit 102 in step S2, the task estimation unit 1051 identifies the task with the highest probability as the task of the worker based on the probability of the hand shape (hand joint position) performing each of the multiple tasks estimated in step S2 and the probability of the object recognized in step S4 for each of the multiple video data extracted in step S3.

[0033] In step S6, the task identification unit 105 displays the task identified in step S5 on a display device (not shown) of the work analysis device 1. If the tool (object) recognized in step S4 is not registered in the task table stored in the task memory unit 204, the task identification unit 105 displays a message such as "The task could not be identified" on the display device (not shown) of the work analysis device 1.

[0034] As described above, the work analysis device 1 according to the first embodiment estimates joint position information of a worker from video data including the worker's work, estimates movement information of the worker based on the estimated joint position information of the worker, extracts a range of video data relating to an object associated with the movement information from the video data based on the estimated movement information of the worker, recognizes the object from the extracted range of video data, and identifies the work of the worker from the recognized object. This enables the work analysis device 1 to recognize objects from images and classify tasks with a small amount of calculation. Furthermore, the work analysis device 1 does not require an expensive GPU or the like and can be implemented on an inexpensive device. Furthermore, the task classification model is easy to interpret, allowing users to use the task analysis device 1 with ease. Furthermore, if there is a problem with the accuracy of task classification, for example, it is possible to separate the problem into whether the accuracy of object recognition is low or the accuracy of detecting characteristic hand joint positions is low, making it easy to expand and improve the classification model. The first embodiment has been described above.

[0035] Next, a second embodiment will be described. In the first embodiment, joint position information of a worker is estimated from video data including the worker's work, motion information of the worker is estimated based on the estimated joint position information of the worker, a range of video data relating to an object associated with the motion information is extracted from the video data based on the estimated motion information of the worker, the object is recognized within the extracted range of video data, and the worker's work is identified from the recognized object. In contrast, the second embodiment detects an object from video data including the worker's work, estimates joint position information of the worker, detects whether the worker's joint position has entered or exited an image area including the detected object based on the estimated joint position information of the worker, extracts a range of video data relating to the detected object from the video data based on the detection result, performs object recognition within the extracted range of video data, and, if the object cannot be recognized within the range of video data, periodically performs object detection to determine the worker's work based on changes in the object's coordinates. This differs from the first embodiment in that As a result, the work analysis device 1A of the second embodiment can recognize objects from images and classify works with a small amount of calculation. The second embodiment will be described below.

[0036] Second Embodiment Fig. 7 is a functional block diagram showing an example of the functional configuration of the work analysis system according to the second embodiment. Elements having the same functions as those of the work analysis system 100 in Fig. 1 are given the same reference numerals, and detailed descriptions thereof will be omitted. As shown in FIG. 7, the work analysis system 100 includes a work analysis device 1A and a camera 2. The camera 2 has the same functions as the camera 2 in the first embodiment.

[0037] <Work analysis device 1A> 7, the work analysis device 1A includes a control unit 10a and a storage unit 20a. The control unit 10a also includes a joint position estimation unit 101, a motion estimation unit 102, an image clipping unit 103a, an object recognition unit 104a, a work identification unit 105, an object detection unit 106, an object area entry / exit detection unit 107, and an object detection activation unit 108. The work identification unit 105 also includes a work estimation unit 1051a.

[0038] The storage unit 20a is a storage device such as a ROM or HDD. The storage unit 20a stores an operating system and application programs executed by the control unit 10a (described later). The storage unit 20a also includes a video data storage unit 201, an action storage unit 202, an object positional relationship storage unit 203, a task storage unit 204, and an object coordinate storage unit 205. The video data storage unit 201, the action storage unit 202, the object positional relationship storage unit 203, and the work storage unit 204 store data equivalent to that of the video data storage unit 201, the action storage unit 202, the object positional relationship storage unit 203, and the work storage unit 204 in the first embodiment. The object coordinate storage unit 205 stores coordinates in the image coordinate system of a tool (object) detected from video data by the object detection unit 106, which will be described later.

[0039] The control unit 10a includes a CPU, a ROM, a RAM, a CMOS memory, and the like, which are configured to be able to communicate with each other via a bus, and are well known to those skilled in the art. The CPU is a processor that controls the entire work analysis device 1A. The CPU reads system programs and application programs stored in ROM via the bus and controls the entire work analysis device 1A in accordance with the system programs and application programs. As a result, as shown in FIG. 7, the control unit 10a is configured to realize the functions of a joint position estimation unit 101, a movement estimation unit 102, an image cropping unit 103a, an object recognition unit 104a, a work identification unit 105, an object detection unit 106, an object area entry / exit detection unit 107, and an object detection activation unit 108. The work identification unit 105 is also configured to realize the function of a work estimation unit 1051a.

[0040] The joint position estimation unit 101, the movement estimation unit 102, and the task identification unit 105 have the same functions as the joint position estimation unit 101, the movement estimation unit 102, and the task identification unit 105 in the first embodiment.

[0041] Similar to the image cropping unit 103 of the first embodiment, the image cropping unit 103a crops out a range of video data relating to an object (tool) associated with the motion information from the video data based on the motion information estimated by the motion estimation unit 102. Furthermore, the image cropping unit 103a crops out a range of video data relating to an object (tool) detected by the object detection unit 106 (described later) from the video data based on the detection result by the object area entry / exit detection unit 107 (described later).

[0042] Similar to the object recognition unit 104 of the first embodiment, the object recognition unit 104a recognizes an object (tool) within the range of video data cut out by the image cutout unit 103a. Furthermore, the object recognition unit 104a recognizes an object (tool) within the range of video data cut out by the image cutout unit 103a based on the detection result by the object area entry / exit detection unit 107, which will be described later.

[0043] The work estimation unit 1051a identifies the work based on a change in the coordinates of the tool (object) detected by the later-described object detection unit 106. The operation of the work estimation unit 1051a will be described later.

[0044] The object detection unit 106 detects a tool (object) from video data including the work of a worker. FIG. 8 is a diagram showing an example of video data including work by a worker. In the video data shown in FIG. 8, a caliper is placed on a table but is not being used by a worker. The object detection unit 106 extracts image features such as edge amounts from the entire image of the video data shown in FIG. 8 using a known method. The object detection unit 106 performs a matching process between the extracted image features and image features for each tool (object) pre-stored in the storage unit 20, detects the tool (object) in the video data, and acquires the coordinates of the image coordinate system of the image area (dashed line rectangle) containing the detected tool (object). The object detection unit 106 stores the coordinates of the image coordinate system of the acquired image area (dashed line rectangle) in the object coordinate storage unit 205. The detection process by the object detection unit 106 may be performed only once at the beginning.

[0045] The object area entry / exit detection unit 107 detects whether the joint position of the worker has entered or exited the image area including the tool (object) detected by the object detection unit 106, based on the joint position information of the worker estimated by the joint position estimation unit 101. Specifically, the object area entrance / exit detection unit 107 detects the position of an image area (dashed rectangle) including the joint position of the worker's hand in the video data of Fig. 8 based on, for example, the joint position information estimated by the joint position estimation unit 101. The object area entrance / exit detection unit 107 determines whether the position of the image area (dashed rectangle) including the joint position of the worker's hand has entered or exited (i.e., overlapped and separated) the position of an image area (chain-line rectangle) including the tool (object) detected by the object detection unit 106. For example, in the case of Fig. 8, the object area entrance / exit detection unit 107 determines that the joint position of the worker has entered or exited the image area of ​​the tool (object) because the image area (dashed rectangle) of the joint position of the worker's hand and the image area (chain-line rectangle) including the tool (object) are separated from each other. 9 and 10, the object area entry / exit detection unit 107 determines that the image area (dashed rectangle) of the worker's hand joint position has entered or exited the image area (dash-dotted rectangle) including the tool (object). In this case, the image cropping unit 103a crops the image area (dash-dotted rectangle) of the object shown in Fig. 10 from the video data, and the object recognition unit 104a recognizes the object (tool) within the range of the video data cropped by the image cropping unit 103a.

[0046] If the object recognition unit 104a cannot recognize the tool (object) detected by the object detection unit 106, the object detection activation unit 108 periodically causes the object detection unit 106 to detect the tool (object). Specifically, for example, if the object recognition unit 104a cannot recognize the tool (object) detected by the object detection unit 106 in the image area indicated by the dashed-dotted rectangle in Fig. 10, the object detection activation unit 108 determines that the worker has started work using the tool (object). Then, the object detection activation unit 108 causes the object detection unit 106 to periodically (e.g., every second) detect the tool (object) from the entire video data in Fig. 10. In this case, if the position of the image area (dashed-dotted rectangle) of the detected tool (object) changes as shown in Fig. 11, the work estimation unit 1051a determines that the worker is performing the work identified by the work identification unit 105 using the tool (object).

[0047] On the other hand, if the position of the image area (dashed-line rectangle) of the tool (object) has not changed (or the tool (object) cannot be detected), is separated from the image area (dashed-line rectangle) of the worker's hand, and the image area (dashed-line rectangle) of the worker's hand is moving, the work estimation unit 1051a determines that the worker has finished using the tool (object). In this case, the object detection activation unit 108 ends the periodic execution of object detection by the object detection unit 106. By doing so, the work analysis device 1A can reduce the number of times the object detection unit 106 executes the object detection process, which is heavy, by using object detection and joint position information to perform the object detection process only when a worker is using a tool (object). Furthermore, the work analysis device 1A can determine whether the work performed by the identified worker is work that uses a tool (object).

[0048] <Analysis process of work analysis device 1A> Next, the operation of the analysis process of the work analysis device 1A according to the second embodiment will be described. 12 is a flowchart illustrating the analysis processing of the work analysis device 1. The flow shown here is repeatedly executed while video data is being input from the camera 2.

[0049] In step S11, the object detection unit 106 detects an object (tool) from the entire video data including the work of the worker.

[0050] In step S12, the joint position estimation unit 101 estimates the joint position information of the worker's hand from the video data.

[0051] In step S13, if the object area entry / exit detection unit 107 determines that the image area of ​​the worker's hand joint position has entered or exited the image area including the object (tool), the image cropping unit 103a crops out the range of video data related to the object (tool) detected in step S11.

[0052] In step S14, the object recognition unit 104a recognizes an object (tool) within the range of the video data extracted in step S13.

[0053] In step S15, the object detection activation unit 108 determines in step S14 whether the object recognition unit 104a was able to recognize the object (tool) detected in step S11. If the object recognition unit 104a was able to recognize the detected object (tool), the object (tool) is in the initial position (not being used), and the process remains in step S15. On the other hand, if the object recognition unit 104a was unable to recognize the detected object (tool), the process proceeds to step S16.

[0054] In step S16, the object detection activation unit 108 causes the object detection unit 106 to periodically execute the detection process for an object (tool).

[0055] In step S17, the work estimation unit 1051a determines whether the position of the image area of ​​the object (tool) detected in step S16 has changed. If the position of the image area of ​​the detected object (tool) has changed, the process proceeds to step S18. On the other hand, if the position of the image area of ​​the detected object (tool) has not changed, the process proceeds to step S19.

[0056] In step S18, the work estimation unit 1051a determines that the worker is performing the work using a tool (object).

[0057] In step S19, if the image area of ​​the object (tool) and the image area of ​​the worker's hand are separated and the image area of ​​the worker's hand is moving, the work estimation unit 1051a determines that the worker is performing work without using an object (tool).

[0058] In step S20, the object detection activation unit 108 ends the object (tool) detection process by the object detection unit 106. Then, the work analysis device 1A ends the analysis process.

[0059] As described above, the work analysis device 1A according to the second embodiment detects objects from video data containing work performed by a worker, estimates the worker's joint position information, detects whether the worker's joint positions have entered or exited an image region containing the detected object based on the estimated worker's joint position information, extracts a range of video data related to the detected object from the video data based on the detection results, performs object recognition on the extracted range of video data, and if the object cannot be recognized within the range of video data, periodically performs object detection on the object to determine the work performed by the worker based on changes in the object's coordinates. This allows the work analysis device 1A to recognize objects from images and classify tasks with a small amount of calculation. Furthermore, the work analysis device 1A does not require an expensive GPU or the like, and can be implemented on an inexpensive device. Furthermore, the task analysis device 1A allows for easy interpretation of task classification models, enabling users to use it with confidence. Furthermore, if there is a problem with the accuracy of task classification, for example, it is possible to separate the problem into whether the accuracy of object recognition is low or the accuracy of detecting characteristic hand joint positions is low, making it easy to expand and improve the classification model. Furthermore, since the object detection process is heavy, the work analysis device 1A can reduce the number of times the object detection process is executed by using object detection and joint position information to perform the object detection process only when the worker is using an object. Furthermore, the work analysis device 1A can determine whether the work performed by the identified worker is work using an object. The second embodiment has been described above.

[0060] The first and second embodiments have been described above, but the work analysis devices 1, 1A are not limited to the above-described embodiments and include modifications and improvements within the scope of achieving the objectives.

[0061] <Variation 1> In the first and second embodiments, the work analysis apparatus 1, 1A is connected to one camera 2, but this is not limiting. For example, the work analysis apparatus 1, 1A may be connected to two or more cameras 2.

[0062] <Variation 2> In the above-described embodiments, the task analysis devices 1 and 1A each have all of the functions, but this is not limiting. For example, a server may include some or all of the joint position estimation unit 101, movement estimation unit 102, image clipping unit 103, object recognition unit 104, task identification unit 105, and task estimation unit 1051 of the task analysis device 1, or some or all of the joint position estimation unit 101, movement estimation unit 102, image clipping unit 103a, object recognition unit 104a, task identification unit 105, task estimation unit 1051a, object detection unit 106, object area entry / exit detection unit 107, and object detection activation unit 108 of the task analysis device 1A. Furthermore, the functions of the task analysis devices 1 and 1A may be realized using a virtual server function or the like on the cloud. Furthermore, the work analysis devices 1, 1A may be configured as a distributed processing system in which the functions of the work analysis devices 1, 1A are distributed among a plurality of servers as appropriate.

[0063] Note that the functions included in the work analysis devices 1 and 1A in the first and second embodiments can be realized by hardware, software, or a combination of these. Here, "realized by software" means that the functions are realized by a computer reading and executing a program.

[0064] The program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0065] In addition, the steps of writing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually.

[0066] In other words, the work analysis device of the present disclosure can take on a variety of different embodiments having the following configurations.

[0067] (1) The work analysis device 1 of the present disclosure is a work analysis device that analyzes the work of a worker, and includes: a joint position estimation unit 101 that estimates joint position information of a worker from video data including the work of the worker; a movement estimation unit 102 that estimates movement information of the worker based on the joint position information estimated by the joint position estimation unit 101; an image cropping unit 103 that crops out a range of video data relating to an object associated with the movement information from the video data based on the movement information estimated by the movement estimation unit 102; an object recognition unit 104 that recognizes objects within the range of the video data cropped by the image cropping unit 103; and a work identification unit 105 that identifies the work of the worker based on the object recognized by the object recognition unit 104. This task analysis device 1 can recognize objects from images and classify tasks with a small amount of calculation.

[0068] (2) In the work analysis device 1 described in (1), when the movement estimation unit 102 estimates the movement information of the worker including multiple movements based on the joint position information, the image cropping unit 103 may crop multiple ranges of video data for each of the estimated multiple movements, the object recognition unit 104 may recognize the object for each range of the multiple video data, and the work identification unit 105 may include a work estimation unit 1051 that estimates the most probable work based on the probability of each of the multiple movements estimated by the movement estimation unit 102 and the probability of the object recognized by the object recognition unit 104 for each range of the multiple video data. In this way, the work analysis device 1 can accurately identify the work of the worker even when the hand shape is unclear.

[0069] (3) The work analysis device 1 described in (1) or (2) may be provided with an action memory unit 202 that stores a rule base or a learned model that outputs worker action information corresponding to the joint position information estimated by the joint position estimation unit 101, an object position relationship memory unit 203 that stores in advance the range of video data that includes an object related to the worker action information based on the worker action information, and a work memory unit 204 that stores in advance a work table that associates objects recognized by the object recognition unit 104 with worker actions. This makes it easier for the task analysis device 1 to interpret the task classification model.

[0070] (4) The work analysis device 1A disclosed herein is a work analysis device that analyzes the work of a worker, and includes an object detection unit 106 that detects an object from video data including the work of the worker, a joint position estimation unit 101 that estimates joint position information of the worker from the video data, an object area entry / exit detection unit 107 that detects whether an image area including the joint positions of the worker has entered or exited an image area including the object detected by the object detection unit 106 based on the joint position information estimated by the joint position estimation unit 101, and an object area entry / exit detection unit 107 that calculates the joint position information of the worker from the video data based on the detection result of the object area entry / exit detection unit 107. an object recognition unit 104a that performs object recognition within the range of the video data cut out by the image cutout unit 103a; an object detection activation unit 108 that periodically causes the object detection unit 106 to detect an object if the object recognition unit 104a cannot recognize an object within the range of the video data; and an operation estimation unit 1051a that identifies an operation based on a change in the coordinates of the object detected by the object detection unit 106 in the video data. This work analysis device 1A can achieve the same effect as (1). [Explanation of symbols]

[0071] 1, 1A work analyzer 10, 10a Control section 101 Joint position estimation unit 102 Motion estimation section 103, 103a Image cutting section 104, 104a Object recognition section 105 Work Specification Department 1051, 1051a Work estimation section 106 Object detection unit 107 Object area entry / exit detection unit 108 Object detection active unit 20, 20a Storage section 201 Video data storage unit 202 Operation memory unit 203 Object positional relationship memory unit 204 Working Memory 205 Object coordinate storage unit 2 Cameras 100 Work Analysis System

Claims

1. A work analysis device that analyzes work performed by a worker, a joint position estimation unit that estimates joint position information of the worker from video data including the work of the worker; a movement estimating unit that estimates movement information of the worker based on the joint position information estimated by the joint position estimating unit; an image cropping unit that crops out a range of video data relating to an object associated with the motion information from the video data based on the motion information estimated by the motion estimating unit; an object recognition unit that recognizes the object within the range of the video data cut out by the image cutout unit; a task identification unit that identifies a task to be performed by the worker based on the object recognized by the object recognition unit; A work analysis device comprising:

2. when the movement estimation unit estimates movement information of the worker including a plurality of movements based on the joint position information, the image cropping unit crops a plurality of ranges of the video data for each of the estimated movements; the object recognition unit recognizes the object for each range of the plurality of pieces of video data; The work specifying unit 2. The task analysis device according to claim 1, further comprising an task estimation unit that estimates the task with the highest probability based on the probability of each of the plurality of tasks estimated by the task estimation unit and the probability of an object recognized for each range of the plurality of pieces of video data by the object recognition unit.

3. a movement storage unit that stores a rule base or a trained model that outputs movement information of the worker corresponding to the joint position information estimated by the joint position estimation unit; an object positional relationship storage unit that stores in advance, based on the operation information of the worker, a range on the video data that includes the object related to the operation information; 3. The work analysis device according to claim 1, further comprising: a work storage unit that stores a work table in which the objects recognized by the object recognition unit are previously associated with the work of the worker.

4. A work analysis device that analyzes work performed by a worker, an object detection unit that detects an object from video data including the work of the worker; a joint position estimation unit that estimates joint position information of the worker from the video data; an object area entry / exit detection unit that detects whether an image area including the joint positions of the worker has entered or exited an image area including the object detected by the object detection unit, based on the joint position information estimated by the joint position estimation unit; an image cropping unit that crops out a range of video data relating to the object detected by the object detection unit from the video data based on a detection result of the object area entry / exit detection unit; an object recognition unit that performs object recognition on the range of the video data cut out by the image cutout unit; an object detection activation unit that periodically causes the object detection unit to detect the object when the object recognition unit cannot recognize the object within the range of the video data; an activity estimation unit that identifies an activity based on a change in coordinates of the object detected by the object detection unit in the video data; A work analysis device comprising:

Images