Work management device, work management system, and work management program

Abstract

To preferably extract each work content even if different works require the same posture.SOLUTION: A work management device 2 includes: a skeleton extraction unit 21 that extracts skeleton data 22 about a person from a photographed video of the person; an object detection unit 234 that detects a prescribed object from the video; a posture detection section 232 that detects a posture of the person from the skeleton data 22; and an analysis unit 23 that estimates a work performed by the person on the basis of a detection result of the prescribed object and a detection result of the posture of the person.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a work management device, a work management system, and a work management program. [Background technology]

[0002] In the industrial field, there is a need for processes such as measuring the cycle time, which is the time it takes for workers to assemble a product, and analyzing the work content to detect missing tasks or non-routine work.Currently, these processes are mainly performed manually.

[0003] Here, to measure work time, some kind of human action indicating the start and end of work was used as the trigger, such as operating a personal computer (PC), scanning a barcode, pressing a button, etc. Alternatively, work time may be measured by extracting data from devices indirectly related to the work, such as the on / off of a drill, the on / off of a switch, or the current value indicating the operation of a device.

[0004] However, these measurement methods require the addition of new equipment and increase the burden on workers. Also, when measuring work time by adding work procedures that are not part of the original work, such work is often not actually performed, making it difficult to obtain accurate data.

[0005] It is also common to record the work status of workers with a video camera and analyze the work status manually. However, analyzing and recording the status of a specific worker from video footage recorded over a long period of time takes a long time. This requires a lot of labor costs and can only process a limited range of data.

[0006] In recent years, it has become common to record the working status of a worker with a video camera and analyze the working status with an information processing device. Patent Document 1 describes an invention in which a motion information registration unit identifies the motion content indicated by motion information, which is skeleton information similar to the target information extracted by a skeleton extraction unit, as the motion content being performed by the target person. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6777819 Summary of the Invention [Problem to be solved by the invention]

[0008] In production sites, different tasks may be performed in similar postures, such as tightening screws on a product and assembling the product. In such cases, it is difficult to distinguish between them using only the skeleton and posture extracted from the video, which makes it impossible to estimate the work content. Therefore, an object of the present invention is to suitably estimate the content of each task even when a person performs a plurality of different tasks in similar postures. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the work management device of the present invention has a skeleton extraction unit that extracts skeleton data of a person from a video of the person, an object detection unit that detects a predetermined object from the video, a posture detection unit that detects the posture of the person from the skeleton data, and an analysis unit that estimates the work of the person based on the detection result of the predetermined object and the detection result of the posture of the person. The object detection unit detects the predetermined object within a predetermined detection range determined from the skeletal data, and in detecting the object, the object detection unit filters the detection results by majority vote of the detection results of the object in the frames in the video. It is characterized by:

[0010] The work management system of the present invention includes a skeleton extraction unit that extracts skeleton data of a person from a video of the person, an object detection unit that detects a predetermined object from the video, a posture detection unit that detects the posture of the person from the skeleton data, and an analysis unit that estimates the work of the person based on the detection results of the predetermined object and the detection results of the posture of the person. The object detection unit detects the predetermined object within a predetermined detection range determined from the skeletal data, and in detecting the object, the object detection unit filters the detection results by majority vote of the detection results of the object in the frames in the video. It is characterized by:

[0011] The work management program of the present invention is for causing a computer to execute the following steps: extracting skeletal data of a person from a video of the person; detecting a predetermined object from the video; detecting the posture of the person from the skeletal data; and estimating the work of the person based on the detection results of the predetermined object and the detection results of the posture of the person. wherein the step of detecting the predetermined object includes detecting the predetermined object within a predetermined detection range determined from the skeleton data, and filtering the detection results by majority vote of the detection results of the object in the frames of the video. is. Other means will be described in the detailed description of the invention. [Effects of the Invention]

[0012] According to the present invention, even if a person performs a plurality of different tasks in similar postures, it is possible to suitably estimate the content of each task. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a configuration diagram of a work management device according to an embodiment of the present invention; [Figure 2] FIG. 10 is a diagram illustrating an example of an image input to the work management device. [Figure 3] FIG. 10 is a diagram showing an example of skeleton data extracted from a video to be input to the work management device. [Figure 4] FIG. 2 is a diagram illustrating an example of skeletal data. [Figure 5] FIG. 10 is a diagram illustrating an example of a region model. [Figure 6] 10 is a time chart showing the working posture determined by the work management device and the work content of the worker. [Figure 7A] 10 is a flowchart showing a work management process. [Figure 7B]10 is a flowchart showing a work management process. [Figure 8] 10 is a flowchart showing a process of associating a working posture with a driver. [Figure 9] FIG. 10 is a diagram showing a real-time screen displayed by the work management device. [Figure 10] FIG. 10 is a diagram showing a history data screen displayed by the work management device. [Figure 11] FIG. 10 is a diagram showing a recognition report screen displayed by the work management device. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a configuration diagram of a work management device 2 according to this embodiment. The following describes an example in which this work analysis system is introduced into an assembly site in the manufacturing industry and applied to analyze the work of workers assembling small personal computers.

[0015] The work analysis system is centered around a work management device 2 and includes a camera 1, a storage device 11, the work management device 2, a monitor 31, and a storage device 32. Each of these devices in the work analysis system is connected to a network such as Ethernet (registered trademark), USB (Universal Serial Bus), or any other suitable hardware interface.

[0016] Camera 1 captures an image of a worker as a subject. The image captured by camera 1 is recorded in storage device 11. Monitor 31 and storage device 32 are the output destinations of the analysis results of work management device 2. In other words, work management device 2 may receive input of image data of a worker currently performing a task, or may receive input of image data of a worker who performed a task in the past.

[0017] The work management device 2 is, for example, a computer system such as an on-premise server or a cloud server. The work management device 2 is configured as a computer having a CPU (Central Processing Unit), memory, a storage means (storage unit) such as a hard disk, and a network interface. In this computer, the CPU executes a program (also called an application or an app for short) loaded into the memory, thereby operating a control unit (control unit) made up of each processing unit.

[0018] The work management device 2 executes a program on a computer system to configure a skeleton extraction unit 21, an analysis unit 23, and an output unit 28. The analysis unit 23 further includes a region detection unit 231, a posture detection unit 232, a background detection unit 233, and an object detection unit 234.

[0019] The analysis unit 23 refers to the region model 24, the posture model 25, the object model 26, the background model 291, and the combination model 292 to estimate the work content being performed by the worker.

[0020] The area detection unit 231 detects the overlap between a predetermined area and the worker's skeleton by referring to the area model 24. The posture detection unit 232 detects the posture of the worker in the video data by referring to each posture stored in the posture model 25 and the skeleton data 22 extracted from the video. The posture model 25 stores the correspondence between each human posture and the skeleton data for that posture.

[0021] Background detection unit 233 detects the background from the video data by referring to background model 291. Background model 291 stores the correspondence between each structure in the background and the appearance data of that structure.

[0022] The object detection unit 234 detects objects from the video data using the object model 26. The object model 26 stores the correspondence between objects such as tools and the appearance data of the objects. In addition to the results detected by these detection units, the analysis unit 23 estimates the work content being performed by the worker by combining the detection results with reference to the combination model 292. The combination model 292 stores the correspondence relationship between each work content being performed by the worker and the detection results of each detection unit that indicate that work content.

[0023] Each of these configured processing units accesses the skeleton data 22 or the estimation result data 27 stored on a nonvolatile memory such as a hard disk.

[0024] The skeleton extraction unit 21 extracts skeleton data 22 of a person captured in video data input from the camera 1 or the storage device 11. The analysis unit 23 receives the video data for analysis and the skeleton data for analysis 22 as input, and obtains estimation result data 27 that estimates the work to be performed by the worker. The output unit 28 outputs the estimation result data 27 to a monitor 31 and a storage device 32, which are external devices.

[0025] The object detection unit 234 uses a machine learning model to detect objects from each frame constituting the video data. For the sake of explanation, the detected object is assumed to be a tool used for work, such as a screwdriver. However, the object is not limited to any particular object / pattern that appears in the image. The object detection unit 234 detects the position and type of object from frame data 4 in FIG. 2 (described later). The object detection unit 234 may detect the object not only over the entire frame, but also within a region based on a specified skeleton. For example, the object detection unit 234 may detect an object within a range that is within the worker's reach and can be grasped by the worker. Specifically, the object detection unit 234 of this embodiment detects a specified object, i.e., a screwdriver, within a detection range near both wrists determined from the skeleton data 22. By limiting the range, the object detection unit 234 can speed up object detection and reduce erroneous object detection.

[0026] The object detection unit 234 uses the results of machine learning of the object model 26 to obtain the coordinates of the detected object and a score indicating the likelihood. The object detection unit 234 excludes detection results with clearly low scores from the results using a predetermined threshold. The object detection unit 234 also determines as detected an object if the number of consecutive detections of the object in multiple consecutive image frames is equal to or exceeds a threshold, and filters the detection results by majority voting, etc., thereby suppressing false detections and preventing the detection from flickering from frame to frame.

[0027] The object detection unit 234 detects objects within a region based on a specified skeleton, such as detecting an object held by a worker. However, parts of an object are often hidden by the worker's body or workspace. When a part of an object is hidden, detection accuracy decreases, making it difficult to measure work time. Therefore, when the object detection unit 234 detects, for example, a Phillips screwdriver or a flathead screwdriver, it estimates the time spent in the same working posture as the work of tightening a screw. The position of the extracted screwdriver is superimposed on the image. The work time is then output as a graph.

[0028] FIG. 2 is a diagram showing an example of an image input to the work management device 2. As shown in FIG. Frame data 4 is generated for each image frame in a moving image that includes a person. The person in frame data 4 is holding an electronic device in his or her hand, but has not yet picked up a driver.

[0029] FIG. 3 is a diagram showing an example of a skeleton 41 extracted from a video image input to the work management device 2, and a part-removing area 42, a tool area 43, and a finished product storage area 44 that are set in advance. Skeleton 41 is the result of skeleton information of a person extracted from frame data 4 by skeleton extraction unit 21 superimposed on this frame data 4, with each feature point shown connected by a line. Skeleton extraction unit 21 can use a known technique for acquiring skeleton information, such as OpenPose (URL=https: / / github.com / CMU-Perceptual-Computing-Lab / openpose).

[0030] The part-taking area 42 is an area where parts are placed, superimposed on the frame data 4. When the part-taking area 42 overlaps with the predetermined skeleton data 22, the analysis unit 23 estimates that the worker's working status is "part-taking." The tool area 43 is an area where a tool is placed, superimposed on the frame data 4. When a tool is present in the tool area 43, the analysis unit 23 estimates that the worker does not yet have the tool.

[0031] The finished product storage area 44 is an area where finished products are stored, superimposed on the frame data 4. When the finished product storage area 44 overlaps with the predetermined skeleton data 22, the analysis unit 23 estimates that the worker's work status is "completed."

[0032] FIG. 4 is a diagram showing an example of the skeleton data 22. As shown in FIG. The skeleton data 22 includes a number column, a feature point column, an X-axis coordinate column, a Y-axis coordinate column, and a score column. Each row stores feature points such as the joints of a person. In the number column, a unique number is assigned to each feature point, for example, 0 for the nose, 1 for the right shoulder, 2 for the right elbow, etc. The feature point column stores the name of the feature point. The X-axis coordinate column stores the X-axis coordinate value of the feature point in the frame data 4. The Y-axis coordinate column stores the Y-axis coordinate value of the feature point in the frame data 4. The score column stores a numerical value representing the likelihood of the feature point.

[0033] FIG. 5 is a diagram showing an example of the area model 24. As shown in FIG. The area model 24 includes an area label column, a feature point number column, a judgment logic column, and a polygon coordinate column. The area label column is a column that stores the name of this area. The feature point number column is a column that stores one or more feature point numbers of the skeleton data 22. When a feature point specified in the feature point number column falls within the area specified in the polygon coordinate column, the analysis unit 23 detects that the feature point has entered the area.

[0034] The determination logic column indicates the logic for determining whether a feature point having one of these numbers has entered this region when there are multiple feature points in the feature point number column. The polygon coordinate column stores the coordinate values ​​of the polygon that represents this region.

[0035] For example, the first row of the area model 24 is the part-picking area 42 in Figure 3, and when any (logical OR) of the feature point numbers (#3 indicates the right wrist, #6 indicates the left wrist) of the worker's skeletal data 22 exists within the polygon (quadrilateral) coordinate data (four vertex coordinates), it is recognized that the worker has picked up a part for the computer being assembled.

[0036] Note that the "logical product" of the judgment logic indicates a judgment based on the logical product of the feature point numbers (for example, both hands), and the "logical sum" indicates a judgment based on the logical sum of the feature point numbers (for example, one hand). In other words, when one wrist of the worker enters the parts-taking area 42, an area judgment is made that "hand enters the parts-taking area."

[0037] The second row of the area model 24 is the finished product storage area 44 in Figure 3, and when any (logical OR) of the feature point numbers (#3 indicates the right wrist, #6 indicates the left wrist) of the worker's skeletal data 22 exists within the polygon (quadrilateral) coordinate data (four vertex coordinates), it is recognized that the worker has stored the assembled finished product.

[0038] The third row of the area model 24 indicates that when both (logical product) of the feature point numbers (#3 indicates the right wrist, #6 indicates the left wrist) of the worker's skeletal data 22 exist within the polygon (rectangle) coordinate data (four vertex coordinates) as the tool area 43 in Figure 3, it is recognized that the worker has taken the screwdriver or put it back.

[0039] FIG. 6 is a time chart showing the working posture determined by the work management device 2 and the work content of the worker. The bar graphs to the right of "posture" indicate that the postures of the worker detected by the posture detection unit 232 from time t1 to t4 and from time t5 to t6 are working postures.

[0040] The bar graph to the right of "Driver Detected" indicates that the object detection unit 234 detected a driver near the right or left wrist of the worker from time t2 to t3. The bar graph to the right of "Work" shows the work content of the worker estimated by the analysis unit 23. From time t1 to t4, the work is estimated to be screw tightening. This is because a screwdriver was detected near the worker's wrist during part of the time period from t1 to t4. In contrast, from time t5 to t6, no screwdriver was detected, so it is not estimated to be screw tightening.

[0041] 7A and 7B are flowcharts showing the work management process. The skeleton extraction unit 21 acquires image frames constituting the video data (step S11), and then acquires skeleton data 22 of the worker in the acquired image frames (step S12).

[0042] Thereafter, the analysis unit 23 performs detection processing in parallel. Specifically, the region detection unit 231 detects a region in the image frame based on the region model 24 (step S13). The posture detection unit 232 detects a posture from the skeleton data 22 based on the posture model 25 (step S14). Here, the posture detection unit 232 calculates a score (likelihood) of the detected posture. If the score is equal to or greater than a threshold, the posture is considered to have been detected, and a character string (label) indicating the detection is output.

[0043] The background detection unit 233 detects the background from the image frame based on the background model 291 (step S15). Here, the background detection unit 233 calculates a score (likelihood) of the detected background. If the score is equal to or greater than a threshold, the background is deemed to have been detected, and a character string (label) indicating the detection is output.

[0044] The object detection unit 234 detects an object from the image frame based on the object model 26 and the skeleton data 22 (step S16). Here, the object detection unit 234 calculates a score (likelihood) of the detected object. If the score is equal to or greater than a threshold, the object is deemed to have been detected, and a character string (label) indicating the detection is output. For object detection, because feature points are defined in the object model 26, the object detection range is set according to the coordinates of the skeleton data 22 extracted in advance, and detection is performed. When these detection processes are completed, the process proceeds to step S17.

[0045] In step S17, the analysis unit 23 performs a filter process based on the number of past detections to suppress flickering of the detected region, posture, background, and object. Then, the analysis unit 23 calculates the detection times of the region, posture, background, and object (step S18). Then, the analysis unit 23 estimates the work based on these calculated detection times. Specifically, the analysis unit 23 performs a process of combining these detection results (step S19).

[0046] In the process of combining the detection results, the analysis unit 23 performs calculations by combining the calculated detection results based on data defined as a combination model 292, and estimates the work content being performed by the worker.

[0047] The combination model 292 stores, for example, a combination of the logical product of the detection of a hand entering the parts-removing area and the detection of the posture of the person taking the part, and the start of the work. Furthermore, the combination model 292 stores a combination of the logical product of the detection of the working posture and the detection of the screwdriver near the wrist, and the screw tightening work.

[0048] For example, the analysis unit 23 performs a logical AND between the detection of the hand entering the part-picking area where the parts are placed in area detection and the detection of the hand being in a position to pick up the parts in posture detection, and if the result is true, it determines that the start work is being performed.

[0049] FIG. 8 is a flowchart showing the process of associating a working posture with a driver. The analysis unit 23 determines whether the object detection unit 234 has detected the driver (step S30). If the object detection unit 234 has not detected the driver (No), the analysis unit 23 ends the processing of Fig. 8, and if the object detection unit 234 has detected the driver (Yes), the analysis unit 23 proceeds to step S31.

[0050] Next, the analysis unit 23 determines whether the posture detection unit 232 has detected a working posture (step S31). If the analysis unit 23 has not detected a working posture (No), it ends the processing in FIG. 8, and if the analysis unit 23 has detected a working posture (Yes), it proceeds to step S32.

[0051] In step S32, the analysis unit 23 determines that the work time is the same as the detection time of the working posture as the screw tightening work time, and ends the processing in Fig. 8. In addition to calculating the presence or absence of detection by logical operation on the detected results, the analysis unit 23 can also perform logical OR or AND on the work time.

[0052] It should be noted that the object detected by the object detection unit 234 is not limited to a screwdriver, but may be a brush, an air gun, a soldering iron, a wrench, etc. The work content estimated by the analysis unit 23 is not limited to screw tightening, but may be painting work when a brush or air gun is detected, soldering work when a soldering iron is detected, bolt tightening work when a wrench is detected, etc.

[0053] Returning to Figure 7B, the explanation will be continued. The analysis unit 23 determines whether the work state estimated from the latest image frame is a started work (step S20). If it is a started work (Yes), the analysis unit 23 starts measuring the work time (step S21) and proceeds to step S24. If it is not a started work (No), the analysis unit 23 proceeds to step S22.

[0054] In step S22, the analysis unit 23 determines whether the work status estimated from the latest image frame is a completed work (step S22). If it is a completed work (Yes), the analysis unit 23 ends measurement of the work time (step S23) and proceeds to step S24. If it is not a completed work (No), the analysis unit 23 proceeds to step S24.

[0055] In step S24, the analysis unit 23 checks the work order. For example, the analysis unit 23 may record, as a work attribute, any work that deviates from the predetermined work order. This allows the analysis unit 23 to distinguish and display work that follows the predetermined work order from work that deviates from the predetermined work order.

[0056] Then, the analysis unit 23 outputs the intermediate result (step S25). Then, the analysis unit 23 determines whether or not there are any unprocessed frames (step S26). If there are any unprocessed frames (Yes), the analysis unit 23 returns to the processing of step S11, and if all frames have been processed (No), the analysis unit 23 proceeds to step S27. In step S27, the analysis unit 23 outputs the estimation result data 27, and then ends the processing of FIGS. 7A and 7B. 7A and 7B, the analysis unit 23 estimates the work being performed by the worker based on the driver detection result and the worker posture detection result. This makes it possible to appropriately extract the details of each work even if the worker performs multiple different work tasks in similar postures.

[0057] FIG. 9 is a diagram showing a real-time screen 5 displayed by the work management device 2. As shown in FIG. 9 is a screen that shows the execution, interruption, and stop of the analysis, as well as the analysis results. The output unit 28 displays this real-time screen 5 on the monitor 31. The real-time screen 5 includes an execution button 51, a pause button 52, a stop button 53, a video display area 56, a status display field 54, a status cumulative time graph 55, and a status time chart 57.

[0058] The video display area 56 is an area where the analysis results are superimposed on the input video data. The execute button 51 is used to execute the analysis. The pause button 52 is used to pause the execution of the analysis. The stop button 53 is used to stop the execution of the analysis.

[0059] The status display column 54 shows the current analysis status, displaying the start and end of work and the status of major work. The status cumulative time graph 55 is a bar graph showing the cumulative time of work in the video being analyzed. The status time chart 57 shows the results of the most recent analysis in a time chart.

[0060] FIG. 10 is a diagram showing the history data screen 6 displayed by the work management device 2. As shown in FIG. The history data screen 6 includes time charts showing the analysis results. The state time chart 60 is a time chart showing a state determined comprehensively based on the score (likelihood) of the analysis result at each time.

[0061] The "start" time chart 61 is a time chart showing a state determined based on the score of the "start" state of the subject in the video at each time. Here, the analysis unit 23 determines that the score of the "start" state is high when the subject's skeleton overlaps with the part-removing area 42 and work has not yet started.

[0062] The "remove parts" time chart 62 is a time chart showing a state determined based on the score of the "remove parts" state of the subject in the video at each time point. Here, the analysis unit 23 determines that the score of the "start" state is high when the subject's skeleton overlaps with the part-removing area 42 and the work has started.

[0063] The "driver in use" time chart 63 is a time chart showing the state determined based on the score of the "driver in use" state of the subject of the video at each time.

[0064] The "driver unused" time chart 64 is a time chart showing the state determined based on the score of the "driver unused" state of the subject of the video at each time. The "screw tightening" time chart 65 is a time chart showing the state determined based on the score of the "screw tightening" state of the subject of the video at each time.

[0065] The "right facing" time chart 66 is a time chart showing a state determined based on the score of the "right facing" state of the subject of the video at each time. The "left facing" time chart 67 is a time chart showing a state determined based on the score of the "left facing" state of the subject in the video at each time.

[0066] The "end" time chart 68 is a time chart showing the state determined based on the score of the "end" state of the subject of the video at each time. The "Placing a part" time chart 69 is a time chart showing the state determined based on the score of the "Placing a part" state of the subject of the video at each time.

[0067] FIG. 11 is a diagram showing the recognition report screen 7 displayed by the work management device 2. As shown in FIG. The recognition report screen 7 displays a legend 70, a label 71, a stacked bar graph 72, a location column 73, and a date and time column 74.

[0068] The stacked bar graph 72 shows the cumulative time of work based on the posture recognized by the analysis unit 23. Here, the four stacked bar graphs 72 show the cumulative time of each work in each time period: from 9:00 to 10:30, 10:50 to 12:20, 13:20 to 14:50, and 15:20 to 16:50.

[0069] The label 71 is used to select the work to be displayed in the stacked bar graph 72. The legend 70 indicates the relationship between the display mode displayed in the stacked bar graph 72 and the work that the display mode indicates. The location column 73 indicates the location that is the subject of the recognition report screen 7. The date and time column 74 indicates the date and time that is the subject of the recognition report screen 7. Here, four stacked bar graphs indicate the cumulative work time for each day. The bar graphs can be displayed by specified unit, such as by location or by date.

[0070] According to this recognition report screen 7, the work manager can easily grasp the cumulative work time of each worker.

[0071] <<Variation>> The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. It is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0072] The above-described configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above-described configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a storage medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0073] In each embodiment, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0074] 1 camera 11 Storage device 2 Work management device 21 Skeleton Extraction 23 Analysis Department 231 Area detection unit 232 Attitude detection unit 233 Background detection unit 234 Object detection unit 24 Area Model 25 Posture Model 26 Object Model 22 Skeletal data 27 Estimation result data 28 Output section 31 Monitor 32 Storage device 4 Frame Data 41 Skeleton 42 Parts Removal Area 43 Tool area 44 Finished product storage area

Claims

1. a skeleton extraction unit that extracts skeleton data of a person from a video of the person; an object detection unit that detects a predetermined object from the video; a posture detection unit that detects a posture of the person from the skeleton data; an analysis unit that estimates an operation of the person based on the detection result of the predetermined object and the detection result of the posture of the person; and the object detection unit detects the predetermined object within a predetermined detection range determined from the skeleton data; the object detection unit filters the detection results by majority vote of the detection results of the object in the frames in the video, A work management device characterized by:

2. The analysis unit calculates a cumulative time of the estimated work. The work management device according to claim 1 .

3. the object detection unit filters the detection result based on the number of consecutive detections of the object in frames in the video, The work management device according to claim 1 .

4. a skeleton extraction unit that extracts skeleton data of a person from a video of the person; an object detection unit that detects a predetermined object from the video; a posture detection unit that detects a posture of the person from the skeleton data; an analysis unit that estimates the operation of the person based on the detection result of the predetermined object and the detection result of the posture of the person; and the object detection unit detects the predetermined object within a predetermined detection range determined from the skeleton data; the object detection unit filters the detection results by majority vote of the detection results of the object in the frames in the video, A work management system characterized by:

5. On the computer, A step of extracting skeletal data of a person from a video of the person; detecting a predetermined object from the video; detecting a posture of the person from the skeletal data; a step of estimating an operation of the person based on the detection result of the predetermined object and the detection result of the posture of the person; A work management program for executing the step of detecting the predetermined object includes detecting the predetermined object within a predetermined detection range determined from the skeleton data; the step of detecting the predetermined object includes filtering the detection results based on a majority vote of the detection results of the object in the frames of the video; A work management program characterized by:

Images