Work management device, work management system, and work management method

Abstract

A work management device (2) comprises: an image processing unit (202) that processes each frame of a moving image of a person into image data including a person region and an object region; a skeleton extraction unit (21) that extracts skeleton data of the person from the image data; a posture detection unit that detects the posture of the person from the skeleton data; and an analysis unit (23) that estimates the work of the person on the basis of the detection result of the posture of the person.

Description

Work management device, work management system, and work management method

[0001] The present invention relates to a work management device, a work management system, and a work management method.

[0002] In the industrial field, there is a need for processes such as measuring the cycle time, which is the time it takes for workers to assemble a product, and analyzing the work content to detect missing tasks or non-routine work.Currently, these processes are mainly performed manually.

[0003] In the analysis of work content, the measurement of work time was triggered by some kind of human operation that indicates the start and end of the work, such as operating a personal computer (PC), scanning a barcode, pressing a button, etc. Alternatively, work time may be measured by extracting data from devices indirectly related to the work, such as the on / off of a drill, the on / off of a switch, or the current value indicating the operation of a device.

[0004] However, these measurement methods require the addition of new equipment and increase the burden on workers. Also, when measuring work time by adding work procedures that are not part of the original work, such work is often not actually performed, making it difficult to obtain accurate data.

[0005] It is also common to record the work status of workers with a video camera and analyze the work status manually. However, analyzing and recording the status of a specific worker from video footage recorded over a long period of time takes a long time. This requires a lot of labor costs and the computer can only process a limited range of data.

[0006] In recent years, the working status of a worker has been recorded by a video camera, and the working status has been analyzed by an information processing device. Patent Document 1 describes an invention in which a motion information registration unit identifies, as the motion content being performed by a subject, the motion content indicated by motion information, which is skeleton information similar to the target information extracted by a skeleton extraction unit.

[0007] Patent No. 6777819

[0008] In the information processing devices described above, if the input image size is large and high resolution, the burden of calculation processing increases, and real-time processing may not be possible. However, if the input image size is reduced, the number of pixels decreases, resulting in insufficient information volume and making image recognition impossible. Therefore, an object of the present invention is to enable a device that estimates the work performed by a person from an input image to process high-resolution images in real time.

[0009] In order to solve the above-mentioned problems, the work management device of the present invention is characterized by having an image processing unit that processes each frame of a video of a person into image data including an area in which the person is desired to be recognized and an area in which an object is desired to be recognized, a skeleton extraction unit that extracts skeletal data of the person from the image data, a posture detection unit that detects the posture of the person from the skeletal data, and an analysis unit that estimates the work of the person based on the detection result of the posture of the person.

[0010] The work management system of the present invention is characterized by having an image processing unit that processes each frame of a video of a person into image data including an area where the person is desired to be recognized and an area where an object is desired to be recognized, a skeleton extraction unit that extracts skeletal data of the person from the image data, a posture detection unit that detects the posture of the person from the skeletal data, and an analysis unit that estimates the work of the person based on the detection result of the person's posture.

[0011] The work management method of the present invention is characterized by having an image processing unit that processes each frame of a video of a person into image data including an area where the person is to be recognized and an area where an object is to be recognized, a skeleton extraction unit that extracts skeleton data of the person from the image data, a posture detection unit that detects the posture of the person from the skeleton data, and an analysis unit that estimates the work of the person based on the detection result of the posture of the person. Other means will be described in the description of the invention.

[0012] According to the present invention, a device that estimates an activity performed by a person from an input image can process high-resolution images in real time.

[0013] 19 is a configuration diagram of a work management system according to a first embodiment. FIG. 19 is a sequence diagram showing the operation of the work management system. FIG. 19 is a diagram showing an example of image data and skeletal data. FIG. 19 is a diagram showing an example of image data after synthesis. FIG. 20 is a table showing feature point data constituting the skeletal data of FIG. 3. FIG. 20 is a diagram showing a configuration diagram of a processing unit related to an area. FIG. 20 is a diagram showing a processing unit related to a posture. FIG. 20 is a diagram showing a processing unit related to a background. FIG. 20 is a diagram showing a processing unit related to a procedure and an output unit that outputs the processing results. FIG. 21 is a flowchart showing model definition by a background definition unit. FIG. 21 is a diagram showing image data that is the subject of model definition. FIG. 22 is a diagram showing an example of an area model generated from the image data of FIG. 11. FIG. 22 is a diagram showing an example of a background model generated from the image data of FIG. 11. FIG. 21 is a flowchart showing model definition by a posture learning unit. FIG. 22 is a GUI screen diagram in the learning process of the posture detection unit of FIG. 14. FIG. 22 is a diagram showing a posture model generated as a result of the learning process of the posture detection unit of FIG. 14. FIG. 22 is a diagram showing a procedure model that is the learning result by the procedure learning unit. FIG. 23 is a flowchart showing main processing of an analysis unit. FIG. 23 is a flowchart showing subroutine processing of an area detection unit. FIG. 24 is a diagram showing when both hands are recognized in a "parts taking area" as a result of the processing of FIG. 19. FIG. 24 is a diagram showing when both hands are recognized in a "finished product storage area" as a result of the processing of FIG. 19. 28 is a diagram showing image data used in the processing of FIG. 22. FIG. 29 is a diagram of posture data showing inference labels (posture labels) for the image data of FIG. 22. FIG. 30 is a flowchart showing subroutine processing of a background detection unit. FIG. 31 is a diagram showing a case where an unused state by the driver is recognized as a result of the processing of FIG. 25. FIG. 32 is a diagram showing a case where an in-use state by the driver is recognized as a result of the processing of FIG. 25. FIG. 33 is a diagram showing an example of procedure data output by a procedure detection unit. FIG. 34 is a screen diagram showing the procedure data of FIG. 28 in a Gantt chart format. FIG. 35 is a configuration diagram of a work management device relating to a second embodiment. FIG. 36 is a diagram showing an example of a moving image input to the work management device. FIG. 37 is a diagram showing an example of skeletal data extracted from frames of a moving image input to the work management device. FIG. 38 is a diagram showing an example of image data obtained by cutting out a portion to be recognized from image data input to the work management device. FIG. 39 is a diagram showing an example of skeletal data. FIG. 39 is a diagram showing an example of an area model.1 is a time chart showing the work content of a worker. FIG. 1 is a time chart showing the work content of a worker. FIG. 2 is a time chart showing the work content of a worker. FIG. 3 is a time chart showing the work content of a worker. FIG. 4 is a time chart showing the working posture determined by the work management device and the work content of the worker. FIG. 5 is a flowchart showing the work management process. FIG. 6 is a flowchart showing the work management process. FIG. 7 is a flowchart showing the process of associating the working posture with the driver. FIG. 8 is a diagram showing a real-time screen displayed by the work management device. FIG. 9 is a diagram showing a history data screen displayed by the work management device. FIG. 10 is a diagram showing a recognition report screen displayed by the work management device.

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a configuration diagram of a work management system. The following describes an example in which this work management system is introduced into an assembly site in a manufacturing industry and applied to analyzing the work of workers assembling small personal computers. The work management system is centered around a work management device 2, and also includes a video camera 1, a video recorder 13, an input / output device 14, a monitor 31, a storage device 32, and an application device 33.

[0015] Each device in the work management system is connected via a network such as Ethernet (registered trademark), USB (Universal Serial Bus), or any other suitable hardware interface. Each device in the work management system may be configured as a standalone device, or may be realized by executing software on a computer system such as the work management device 2.

[0016] The video camera 1 captures images of the worker as a subject. The video recorder 13 records the video captured by the video camera 1. The input / output device 14 includes a graphic display and a mouse, and displays information to users such as workers and receives instructions from the users. The monitor 31, storage device 32, and application device 33 are the output destinations of the analysis results of the work management device 2 (see FIG. 9 for details).

[0017] The work management device 2 is, for example, a computer system such as an on-premises server or a cloud server. The work management device 2 is configured as a computer having a CPU (Central Processing Unit), memory, a storage means (storage unit) such as a hard disk, and a network interface. In this computer, the CPU executes a program loaded into the memory, thereby operating a control unit (control means) made up of each processing unit.

[0018] The work management device 2 executes a program on a computer system to configure an image processing unit 202, a skeleton extraction unit 21, a model generation unit 211, an analysis unit 23, and an output unit 28. Each of these configured processing units accesses data (model data 212, estimation result data 27) stored on a non-volatile memory such as a hard disk.

[0019] The image processing unit 202 synthesizes an area where a person is to be recognized and an area where an object is to be recognized based on image data 210, which is each frame of a moving image input from the video camera 1 or video recorder 13, and processes the image data into new image data 213. Since the number of pixels of this image data 213 is smaller than the number of pixels of the image data 210, even if the image data 210 has high resolution, the burden of calculation processing is small and real-time processing is excellent.

[0020] The image processing unit 202 extracts an area where a person is to be recognized and an area where an object is to be recognized from image data 210, which is each frame of a moving image, and processes the image data 213 by arranging the area where a person is to be recognized in one side and the area where an object is to be recognized in the other side and combining the two. Since each area is arranged according to the object to be recognized, it is possible to recognize a person from one side of the image data 213 and an object from the other side of the image data 213. This reduces the burden of calculation processing and provides excellent real-time processing capabilities.

[0021] The image processing unit 202 extracts an area where a person is to be recognized and an area where an object is to be recognized from image data 210, which is each frame of a moving image, and processes the extracted image data 213 by adding margins and combining the extracted image data. This makes it possible to suppress false detections in object recognition and skeleton extraction at the joints between the areas.

[0022] Furthermore, the image processing unit 202 cuts out an area where a person is to be recognized and an area where an object is to be recognized from image data 210, which is each frame of the moving image, and processes the image data 213 by resizing and combining the cutouts. Since each area is resized according to the object to be recognized, even if the image data 210 has high resolution, the burden of calculation processing is small, and real-time processing is excellent.

[0023] Furthermore, the image processing unit 202 may extract an area in which a person is to be recognized and an area in which an object is to be recognized from image data 210, which is each frame of a moving image, and process the extracted image data 213 by filtering and combining the extracted image data. This makes it possible to suppress false detections in object recognition and skeleton extraction.

[0024] The image processing unit 202 may use pattern matching to adjust the positions at which the areas where people and objects are to be recognized are cut out from the image data 210, which is each frame of the video. This makes it possible to adjust for deviations in the installation position of the video camera 1 and deviations in the worker's work position.

[0025] The image processing unit 202 may use markers to adjust the positions at which the areas where people and objects are to be recognized are cut out from the image data 210, which is each frame of the video. This makes it possible to adjust for deviations in the installation position of the video camera 1 and deviations in the worker's work position. The image processing unit 202 may be provided with an adjustment unit that displays a screen on which the positions at which the areas where people and objects are to be recognized are cut out from the image data 210, which is each frame of the video, can be manually adjusted. This makes it possible to adjust the areas where people and objects are to be recognized to positions desired by the user.

[0026] The skeleton extraction unit 21 extracts skeleton data 22 based on image data 213 input from the image processing unit 202. The skeleton extraction unit 21 may extract the skeleton data 22 from an area of ​​the image data 213 where a person is desired to be recognized. Even if the image data 210 has high resolution, the burden of calculation processing is reduced, and the skeleton extraction unit 21 can provide processing that is excellent in real-time processing ability.

[0027] The model generation unit 211 receives as input the image data 213 for learning and the skeletal data 22 for learning, generates model data 212, and stores the model data 212 in a non-volatile memory. The model data 212 includes definition data explicitly defined by the user and learned data, which is the result of learning using label data input by the user. The model generation unit 211 basically needs to create the model data 212 once for the task to be analyzed, but may update (improve) the already created model data 212 to improve accuracy. The analysis unit 23 receives as input the image data 213 for analysis and the skeletal data 22 for analysis, and performs inference processing using the model data 212 to obtain estimation result data 27. The output unit 28 converts the estimation result data 27, which is represented by the coordinates of the image data 213, into the coordinates of the image data 210, which is the frame before being processed into the image data 213. The output unit 28 superimposes the converted estimation result data 27 on the image data 210, which is the frame before being processed into image data 213, and outputs it to an external device (a monitor 31, a storage device 32, and an application device 33).

[0028] 2 is a sequence diagram showing the operation of the work management system. In a machine learning stage such as deep learning, image data 210 acquired from video camera 1 (step S101), or image data 210 acquired from video camera 1 (step S102) and recorded in image recorder 13's recording image 32D (step S103), is input to work management device 2. Work management device 2 extracts multiple image portions to be recognized from image data 210 to create a single image data 213, and then executes a learning process (step S112) in response to a learning instruction (step S111) received from the user via input / output device 14, and outputs the result as model data 212.

[0029] In the analysis stage, image data 210 acquired from video camera 1 (step S121), or image data 210 acquired from video camera 1 (step S122) and recorded in recorded image 32E by video recorder 13 (step S123), is input to work management device 2. Work management device 2 extracts multiple image portions to be recognized from image data 210 to create a single piece of image data 213, and then executes analysis processing (step S132) based on model data 212 in response to an analysis instruction (step S131) ​​received from the user via input / output device 14, and outputs the result as estimation result data 27. Note that work management device 2 may extract multiple image portions to be recognized from image data 210 acquired (step S121) to create a single piece of image data 213, and execute analysis processing (step S132) in real time. The work management device 2 may automatically execute the analysis process (step S132) without receiving an analysis instruction from the user (step S131). Then, the output unit 28 of the work management device 2 outputs the estimation result data 27 to the application device 33 or the like by performing an output process (step S141).

[0030] FIG. 3 illustrates an example of image data 210 and skeletal data 22. Image data 210 is generated for each frame of a moving image containing a person. A part-taking area 901, a finished product storage area 902, a tool area 903, and a person area 910 are predefined in the image data 210. The person area 910 is a person area where a person is assumed to be captured. The part-taking area 901, the finished product storage area 902, and the tool area 903 are object areas where an object is assumed to be captured. Skeletal data 22 is the result of the skeleton extraction unit 21 extracting skeletal information of a person from the image data 210. In the skeletal data 22, a number is assigned to each feature point (such as a joint point) of the person (numbers 0 to 9 in the figure). The skeleton extraction unit 21 can use a known skeletal information acquisition technology such as OpenPose (URL: https: / / github.com / CMU-Perceptual-Computing-Lab / openpose).

[0031] The part-removing area 901 is located on the right side of the worker and is an area where the worker removes parts. When the analysis unit 23 detects predetermined skeleton data 22 in the part-removing area 901, it estimates that the worker's working status is "removing parts."

[0032] The tool area 903 is located slightly to the right of the worker in front of him / her, and is an area where the worker takes out tools and places tools that he / she has finished using. When a tool is present in the tool area 903, the analysis unit 23 infers that the worker does not yet have that tool.

[0033] The finished product storage area 902 is located on the left side of the worker and is an area where the worker stores the finished product. The analysis unit 23 estimates that the worker's work status is "completed" when the finished product storage area 902 overlaps with the predetermined skeletal data 22. The person area 910 is located around the worker's chair and is an area for photographing the worker.

[0034] Fig. 4 is a diagram showing an example of image data 213 after synthesis. The image processing unit 202 removes unnecessary portions from the image data 210 of Fig. 3. The image processing unit 202 synthesizes the part-removing area 901, the finished product storage area 902, the tool area 903, and the person area 910 from the image data 210, and processes the image data 213 into new image data 213. By cutting out multiple areas to be recognized from the large-sized image data 210 and combining them into a single image, the size of the image data 213 becomes smaller than the original image data 210.

[0035] 5 is a table showing the feature point data constituting the skeleton data 22 of FIG. 3. This table is composed of the name of the feature point (e.g., nose) and the (x, y) coordinates of the feature point for each feature point number. For example, different numbers are assigned to feature points such as a person's neck, left shoulder, and left elbow. The feature point names and coordinates are the results of the skeleton extraction unit 21 recognizing each joint point from the image data 213 through image recognition.

[0036] The model generation unit 211 and the analysis unit 23 will be described in detail below with reference to Figures 6 to 9. The work management device 2 obtains the final analysis result (4) based on the intermediate analysis results (1) to (3) below. (1) Analysis of the "area" refers to an analysis of whether the body of the worker indicated by the skeletal data 22 is within a predefined area in the image data 213 (see Figure 6 for details).

[0037] (2) "Posture" analysis refers to analyzing the posture of the worker's body as indicated by the skeletal data 22 (see FIG. 7 for details). (3) "Background" analysis refers to analyzing the state of a background area predefined in the image data 213 (see FIG. 8 for details).

[0038] (4) Analysis of "procedure" refers to analyzing what procedure in the assembly work the worker in the image data 213 is performing based on a combination of the analysis results of "area, posture, and background" (see FIG. 9 for details). The output unit 28 may output the final analysis result of (4), or may output at least one of the intermediate analysis results shown in (1) to (3).

[0039] 6 is a block diagram showing a processing unit related to regions. The region definition unit 231a of the model generation unit 211 allows the user to define a region on the image data 210 as polygonal (quadrilateral) coordinate data via the input / output device 14, and saves the definition data as a region model 24 of the model data 212.

[0040] The area detection unit 231 of the analysis unit 23 uses the saved area model 24 and skeletal data 22 to analyze whether or not the feature points of the skeletal data 22 are within the defined area, and outputs the analysis result (person working state) as area data 24a of the estimation result data 27.

[0041] The region definition unit 231a may or may not use machine learning such as deep learning. Machine learning is highly accurate and versatile. However, it requires a huge amount of image data 210 for learning, which is time-consuming. Furthermore, since it requires deep learning technology, it is not available to production management personnel at the manufacturing site. Therefore, instead of using machine learning, the burden on manufacturing site personnel can be reduced by having the user directly define the region model 24.

[0042] 7 is a block diagram showing a posture-related processing unit. The posture learning unit 232a of the model generation unit 211 displays the skeleton data 22 extracted by the skeleton extraction unit 21 and accepts a correct label (posture label) from a user who views the display. The posture learning unit 232a learns combination data of the skeleton data 22 and the posture label, and saves the learning result as a posture model 25 of the model data 212. The posture detection unit 232 of the analysis unit 23 uses the saved posture model 25 and the skeleton data 22 to analyze the posture of the person in the skeleton data 22, and outputs the analysis result (person working state) as posture data 25a of the estimation result data 27.

[0043] FIG. 8 is a block diagram showing a background-related processing unit. The background definition unit 233a of the model generation unit 211 allows the user to define the background region of the image data 210 as polygonal (rectangle) coordinate data via the input / output device 14, and accepts a correct label (background label) from the user upon viewing the image content within the background region. The background definition unit 233a learns combination data of the background region and background label, and stores the learning results as a background model 291 in the model data 212. The background detection unit 233 of the analysis unit 23 uses the stored background model 291 and the image data 210 to analyze the image content within the background region of the image data 213, and outputs the analysis results (equipment operation status) as background data 291a in the estimation result data 27. Based on the image data 210, the image data 213 is generated by extracting and combining the region of the region model 24, the region of the background model 291, and the region where people are primarily photographed.

[0044] FIG. 9 is a block diagram showing a procedure-related processing unit and an output unit 28 that outputs the processing results. Model data 212, including the area model 24, posture model 25, background model 291, and procedure model 297, and estimation result data 27, including the area data 24a, posture data 25a, background data 291a, and procedure data 297a, are stored in the storage unit 200 of the work management device 2. The procedure detection unit 237 combines the estimation result data 27 of the intermediate analysis results of the area data 24a, posture data 25a, and background data 291a to determine the final procedure data 297a (task state) of the worker. Even if one of the three types of intermediate analysis results is incorrectly estimated, the remaining two types are correctly estimated, thereby improving final accuracy.

[0045] The process of determining procedure data 297a by the procedure detection unit 237 requires a procedure model 297, which is model data 212 for determining procedure data 297a from a combination of intermediate model data 212 for each of "region model 24, posture model 25, and background model 291." Therefore, the procedure learning unit 237a displays a combination of intermediate analysis results for each of "region, posture, and background," and accepts a correct label (procedure label) from a user who views the display. The procedure learning unit 237a learns the combination of intermediate analysis results and the procedure label, and saves the learning result as a procedure model 297 of the model data 212. In this way, by combining learning and inference using machine learning techniques, analysis can be performed efficiently in a shorter time.

[0046] The output calculation unit 281 of the output unit 28 receives notification of the estimation result data 27 and executes the calculation process exemplified below to generate data required by the output destination. The HTML output unit 282 converts the estimation result data 27 into HTML format (browser display) and outputs it to the monitor 31. The CSV output unit 283 converts the estimation result data 27 into a CSV format file and outputs it to the storage device 32. The socket communication unit 284 outputs the estimation result data 27 to the application device 33 via socket communication.

[0047] Examples of the model generation unit 211 will be described below with reference to FIGS. 10 to 17. FIG. 10 is a flowchart showing model definition by the background definition unit 233a. In step S301, the background definition unit 233a acquires image data 210 of a frame selected using a GUI (Graphical User Interface). In step S302, the background definition unit 233a accepts input for labeling the selected frame with a background label. In step S303, the background definition unit 233a cuts out image data of a background region defined by polygonal (quadrilateral) coordinate data that is part of the image data 210 of the selected frame. In step S304, the background definition unit 233a stores a combination of the image data of step S303 and the background label of step S302 as learning data for the selected frame.

[0048] In step S305, if there are any unprocessed frames, the background definition unit 233a returns the process to step S301. In step S306, the background definition unit 233a performs machine learning using the learning data of step S306 as input. For machine learning, known techniques such as neural networks and ensemble learning, including deep learning, can be used. In step S307, the background definition unit 233a saves the learning result of step S306 as a background model 291, and the process in FIG. 10 ends. As described above, through the processes of steps S301 to S307, the background definition unit 233a defines the background model 291 from the image data 213.

[0049] FIG. 11 shows image data 210 and 213 that are the targets of model definition. The area definition unit 231a uses the GUI of the input / output device 14 to define a parts taking area 901 and a finished product storage area 902 on the image data 210 for the area model 24. For example, the parts taking area 901 is placed to the right of the worker's chair, and the finished product storage area 902 is placed to the left. The background definition unit 233a uses the GUI of the input / output device 14 to define a tool area 903 on the image data 210 for the background model 291. For example, the driver area, which is a place for placing a screwdriver, is placed to the right of the worker's chair (tool area 903). The image data 213 is a combination of the parts taking area 901, the finished product storage area 902, the tool area 903, and the person area 910. In this way, the image processing unit 202 may be able to acquire and set area information to be analyzed from the model data 212.

[0050] FIG. 12 is a diagram showing an example of an area model 24 generated from the image data 210 of FIG. 11 . The area model 24 is configured by associating area labels, feature point numbers, judgment logic, and polygon (rectangle) coordinate data for each area input in FIG. 10 . For example, the first row of the area model 24 is a "parts-taking area." When both (AND) feature point numbers (4 indicates the right wrist and 7 indicates the left wrist) of the worker's skeletal data 22 are present within the polygon (rectangle) coordinate data (four vertex coordinates, representing the parts-taking area 901 of FIG. 11 ), the worker is recognized as having picked up a part for a personal computer to be assembled. Note that the "AND" in the judgment logic indicates an AND judgment of the feature point numbers (e.g., both hands), and the "OR" indicates an OR judgment of the feature point numbers (e.g., one hand). In other words, if both wrists of the worker are within the parts-taking area, an area judgment is made that "both hands can fit on the right side."

[0051] FIG. 13 is a diagram showing an example of a background model 291 generated from the image data 210 of FIG. 11 . The background model 291 is configured by associating a definition name, polygonal (rectangle) coordinate data input in step S303, and a background label input in step S302. For example, the user associates the background label "unused (with driver)" with the polygonal (rectangle) coordinate data of the tool area 903 in which a driver is placed in the driver storage area. On the other hand, although not shown, the user associates the background label "in use (without driver)" with the polygonal (rectangle) coordinate data of the tool area 903 in which a driver is not placed in the driver storage area. In other words, even if the positions of the areas in the image data 213 indicated by the polygonal (rectangle) coordinate data are the same, different background labels are associated with the image data 210 in which a driver is placed and the image data 210 in which a driver is not placed.

[0052] 14 is a flowchart showing model definition by posture learning unit 232a. In step S311, posture learning unit 232a acquires image data 210 of a frame selected using a GUI (described later in FIG. 15). In step S312, posture learning unit 232a acquires skeletal data 22 of the frame selected in step S311. Then, posture learning unit 232a displays image data 210 and skeletal data 22, and prompts the user to input a posture model based on the displayed content. In step S313, if a posture label (correct label) has not been assigned, posture learning unit 232a returns the process to step S311 and selects another frame.

[0053] In step S314, the posture learning unit 232a stores the combination of the skeleton data 22 of step S312 and the posture label of step S313 as learning data for the selected frame. In step S315, if there is an unprocessed frame, the posture learning unit 232a returns the process to step S301. In step S316, the posture learning unit 232a performs machine learning using the learning data of step S314 as input, as in step S306. In step S317, the posture learning unit 232a saves the learning result of step S316 as posture model 25 (described later in FIG. 16 ).

[0054] 15 is a diagram showing a GUI screen in the learning process of the orientation detection unit 232 in FIG. 14. The user assigns a correct label using the GUI screen 810 of the input / output device 14. First, the user selects an image from the image selection field 812 using the frame advance buttons or slider while viewing a learning image from the image display field 811. The user indicates whether the selected image is "take out from the right," "assemble," "place on the left," or "other" by pressing a button in the correct label input field 813.

[0055] FIG. 16 is a diagram showing a posture model 25 generated as a result of the learning process of the posture detection unit 232 in FIG. 14. The posture detection unit 232 associates the frame number input from the GUI in FIG. 15, the correct label, and the person's skeletal data 22 detected in the frame as a posture model 25 resulting from machine learning. This posture model 25 is used, for example, to learn the working posture of screwdriving and to estimate whether or not a screw is being screwed. Note that FIG. 16 shows a posture model 25 for determining "take out from the right," "assemble," "place on the left," or "other" from the posture of the person in the image displayed in the image display field 811 in FIG. 15.

[0056] FIG. 17 is a diagram showing a procedure model 297, which is the result of learning by the procedure learning unit 237a. The procedure model 297 is a model for outputting the procedure of a worker estimated from a combination of the region model 24, the posture model 25, and the background model 291 as an input model. For example, an assembly task may be composed of the following procedures: The part removal procedure is a procedure for obtaining the parts to be assembled that are located to the right of the worker. The assembly procedure is a procedure for tightening screws using a screwdriver. The part storage procedure is a procedure for placing the completed assembled parts to the left of the worker. For example, using the region model 24 alone can understand "how" a person's hand moved, but it is unclear "what" the person's hand grasped. However, by using the region model 24 and the background model 291 together and noting that there is no screwdriver in the screwdriver storage area in the background region, it becomes clear that the person grasped "a screwdriver." Furthermore, by using the posture model 25 in combination, it is possible to determine whether the driver has picked up or put down the screwdriver based on the angle of the elbow, etc., after determining that the hand has reached out for the screwdriver.

[0057] Below, examples of the analysis unit 23 will be described with reference to FIGS. 18 to 29. FIG. 18 is a flowchart showing the main processing of the analysis unit 23. In step S411, the analysis unit 23 acquires model data 212. In step S412, the analysis unit 23 acquires image data 213 for analysis. In step S413, the analysis unit 23 causes the skeleton extraction unit 21 to extract skeleton data 22 from the image data 213 of step S412. If a region model 24 exists (step S421, Yes), the analysis unit 23 causes the region detection unit 231 to execute an estimation process for region data 24a (step S422, details of which are shown in FIG. 19). If a posture model 25 exists (step S423, Yes), the analysis unit 23 causes the posture detection unit 232 to execute an estimation process for posture data 25a (step S424, details of which are shown in FIG. 22).

[0058] In step S425, if an unprocessed person exists in the image data 210 of step S412, the analysis unit 23 returns the process to step S21. If a background model 291 exists (step S426, Yes), the analysis unit 23 causes the background detection unit 233 to execute estimation processing of background data 291a (step S427, details of which are shown in FIG. 25). In step S431, if an unprocessed frame exists, the analysis unit 23 returns the process to step S412. In step S432, the analysis unit 23 causes the procedure detection unit 237 to estimate a work procedure from the analysis results of steps S422, S424, and S427.

[0059] 19 is a flowchart showing subroutine processing of the area detection unit 231. In step S221, the area detection unit 231 acquires, for each frame, skeletal data 22 of the person detected in that frame. In step S222, the area detection unit 231 acquires one record (one area) from the area model 24. If the coordinates of the feature point numbers constituting the skeletal data 22 in step S221 are within the area acquired in step S222 (step S223, Yes), the area detection unit 231 retains the area label of the record acquired in step S222 (step S224).

[0060] In step S225, if there are any unprocessed records in the area model 24, the area detection unit 231 returns the process to step S222. In step S226, the area detection unit 231 outputs all the results held in step S224 as area data 24a. In step S227, if there are any unprocessed frames, the area detection unit 231 returns the process to step S221.

[0061] Fig. 20 is a diagram showing the recognition of both hands in the "parts taking area" as a result of the processing in Fig. 19. The area detection unit 231 estimates area data 24a including the area label "taking parts from the right" for the parts taking area 901 on the image data 213 in Fig. 11 by satisfying the requirements of the first record (parts taking area) of the area model 24 in Fig. 12 (i.e., both hands fit on the right side).

[0062] Fig. 21 is a diagram showing the recognition of both hands in the "finished product storage area" as a result of the processing in Fig. 19. The area detection unit 231 estimates area data 24a including the area label "parts are stored on the left side" for the finished product storage area 902 on the image data 213 in Fig. 11 by satisfying the requirements of the second record (finished product storage area) of the area model 24 in Fig. 12 (i.e., both hands fit on the left side).

[0063] FIG. 22 is a flowchart showing the subroutine processing of the posture detection unit 232. In step S241, the posture detection unit 232 acquires skeletal data 22 of a person detected in each frame of a video. In step S242, the posture detection unit 232 uses the acquired skeletal data 22 as input and performs machine learning inference using the posture model 25. As a result, a posture label corresponding to the skeletal data 22 is output. Note that if the user determines that the posture label in step S242 is different from the actual posture (inference error) (step S243, Yes), the posture learning unit 232a may accept a correct posture label from the user. Then, the posture learning unit 232a may correct (re-learn) the posture model 25 using a combination of the accepted posture label and the acquired skeletal data 22 as new learning data (step S244).

[0064] In step S245, the posture detection unit 232 holds the output posture label as an inference result. In step S246, if an unprocessed person exists in the frame of step S241, the posture detection unit 232 returns the process to step S243. In step S247, the posture detection unit 232 outputs the inference results held in step S245 for all people existing in the frame as posture data 25a. In step S248, if an unprocessed frame exists, the posture detection unit 232 returns the process to step S241.

[0065] Fig. 23 is a diagram showing image data 210 used in the processing of Fig. 22. The output unit 28 displays the image data 210 in chronological order, along with frame numbers (f10 = 10, f30 = 30, ...) from left to right. In the image data 210 for each frame, lines indicating skeletal data 22 recognized by the skeleton extraction unit 21 are also superimposed on the image of the person. In practice, the detection process is performed using image data 213 that combines the person region and object region of the image data 210.

[0066] 24 is a diagram of posture data 25a showing inferred labels (posture labels) for image data 210 in FIG. 23. The posture detection unit 232 analyzes postures constituting the behavior of the person in image data 213 acquired from the video camera 1 or the like, and outputs the analysis results as posture data 25a. A frame number indicating the detection time is assigned to posture data 25a. This output posture data 25a can be used, for example, to detect working postures related to assembly work procedures at assembly sites in the manufacturing industry, or working postures that place a heavy physical burden on workers and are related to work safety at manufacturing sites in the manufacturing industry.

[0067] FIG. 25 is a flowchart showing subroutine processing of the background detection unit 233. In step S271, the background detection unit 233 acquires image data 210, which is each frame of a moving image. In step S272, the background detection unit 233 acquires one record (one background region) from the background model 291. In step S273, the background detection unit 233 cuts out an image at the position of the background region in step S272 from the image data 213 of step S271. In step S274, the background detection unit 233 uses the image data 210 cut out in step S273 as input and performs machine learning inference using the background model 291. In step S275, the background detection unit 233 retains a background label as the inference result of step S274. In step S276, if there are unprocessed records in the background model 291, the background detection unit 233 returns the process to S272. In step S277, if there is an unprocessed frame, the background detection unit 233 returns the process to step S271. In step S278, the background detection unit 233 outputs the inference results of all background labels.

[0068] 26 is a diagram showing the result of the processing shown in FIG. 25 when the driver recognizes an unused state. The output unit 28 displays "unused" indicated by a background label 903e as background image information related to the human behavior, superimposed on the human behavior image. The output unit 28 also displays skeletal data 22 recognized by the skeleton extraction unit 21 for image data 210, which is each frame of the video, superimposed on the image of the person. Furthermore, the output unit 28 marks and displays feature point data (joint points) that make up the skeletal data 22 (circles in the figure).

[0069] Fig. 27 is a diagram showing the result of the processing in Fig. 25 when the driver recognizes the in-use state. The output unit 28 superimposes "in use" indicated by a background label 903f on the human behavior image as background image information related to the human behavior.

[0070] 28 is a diagram showing an example of procedure data 297a output by the procedure detection unit 237. As described in FIG. 9, the procedure detection unit 237 combines the analysis results of the region data 24a, posture data 25a, and background data 291a to determine the procedure data 297a (work status) of the worker for each frame number. For example, by distinguishing between an unused state ( FIG. 26 ) and an in-use state ( FIG. 27 ) of the driver as background data 291a, it is possible to determine whether the output procedure is being assembled.

[0071] 29 is a screen diagram showing the procedure data 297a of FIG. 28 in Gantt chart format. The procedure detection unit 237 can determine the chronological work procedures (output procedures) from the procedure data 297a because the frame numbers of the procedure data 297a indicate specific times. Therefore, the output unit 28 can clearly show the user the required time for each work procedure by displaying the chronological work procedures in Gantt chart format.

[0072] In the first embodiment described above, the work management device 2 detects a person captured in image data 213 using image recognition such as deep learning and acquires skeletal data 22 of the person. The work management device 2 then performs machine learning on the acquired skeletal data 22 and previously input correct labels as a posture model 25, thereby estimating the posture of the worker at the manufacturing site and identifying the worker's work procedure from the posture. The work management device 2 also improves the accuracy of identifying the work procedure by using a region model 24 for determining the positional relationship with the skeletal data 22 and a background model 291 that indicates the situation in which the person is placed. This allows a simple method to analyze the worker's posture data 25a from image data 213 capturing the worker and identify the worker's procedure data 297a, regardless of knowledge of deep learning or image recognition.

[0073] 30 is a configuration diagram of a work management device 2 according to the second embodiment. The following describes an example in which this work management system is introduced into an assembly site in a manufacturing industry and applied to the analysis of work performed by workers assembling small personal computers.

[0074] The work management system is centered around the work management device 2 and includes a video camera 1, a video recorder 13, a microphone 12, the work management device 2, a monitor 31, and a storage device 32. Each of these devices in the work management system is connected to each other via a network such as Ethernet (registered trademark), a USB (Universal Serial Bus), or any other suitable interface that can be used as a hardware interface.

[0075] Video camera 1 captures images of the worker himself, as well as objects and the background around the worker. Video recorder 13 records the images captured by video camera 1. Microphone 12 records sounds generated at the worker's work site. Monitor 31 and storage device 32 are output destinations for the analysis results of work management device 2. In other words, work management device 2 may receive image data of a worker currently performing a task, or image data of a worker who has performed a task in the past.

[0076] The work management device 2 is, for example, a computer system such as an on-premise server or a cloud server. The work management device 2 is configured as a computer having a CPU (Central Processing Unit), memory, a storage means (storage unit) such as a hard disk, and a network interface. In this computer, the CPU executes a program (also called an application or an app for short) loaded into the memory, thereby operating a control unit (control unit) made up of each processing unit. However, this is not limited thereto, and the work management device 2 may also have a GPU (Graphics Processing Unit), and the GPU may be used for analyzing and displaying the model.

[0077] The work management device 2 executes a program on a computer system to configure an image processing unit 202, a preprocessing unit 201, a skeleton extraction unit 21, an analysis unit 23, and an output unit 28. The analysis unit 23 further includes an area detection unit 231, a posture detection unit 232, a background detection unit 233, an object detection unit 234, a sound detection unit 235, and a movement detection unit 236.

[0078] 31, which will be described later, the image processing unit 202 processes a plurality of regions to be recognized into new image data 213 based on image data 210 that constitutes each frame of a moving image input from the video camera 1 or video recorder 13. Since the number of pixels of this image data 213 is smaller than the number of pixels of the image data 210, even if the image data 210 has high resolution, the burden of calculation processing is small and real-time processing is excellent.

[0079] Returning to Figure 30, the explanation will continue. The sound information recorded by the microphone 12 includes background sounds (for example, the sound of vehicles passing on nearby roads) measured at the work site. The preprocessing unit 201 performs preprocessing using FFT (Fast Fourier Transformation) or the like on the raw sound or voice data to remove background noise and extract the sounds generated at the work site as sound element information. The sound element information extracted by the preprocessing unit 201 is sound data 296.

[0080] The skeleton extraction unit 21 extracts the skeleton data 22 based on the image data 213 input from the image processing unit 202. Note that the skeleton extraction unit 21 may not only extract the skeleton data 22 from the image data 213, but may also detect a person from a predetermined area, for example, by extracting the skeletal data 22 of a worker from the person area 910.

[0081] The analysis unit 23 estimates the work being performed by the worker by referring to the area model 24, the posture model 25, the object model 26, the background model 291, the combination model 292, the sound model 293, and the movement analysis model 294. In the learning stage described above, the analysis unit 23 performs a learning process by adding sound element information in addition to image element information. This learning, for example, involves learning each element and performing a clustering process on six-dimensional data based on six sets of element information (area, posture, background, object, sound, movement) from images and sounds acquired during the same time period. The analysis unit 23 may estimate the work being performed by the worker from the detection results of the area, background, object, sound, movement, and the person's posture. The detection results used for the estimation may be used alone or in combination, and are not limited to these.

[0082] According to the multiple clusters generated by this clustering process, a cluster is generated for each similar task (work process) at the work site. For example, if cluster X is generated for a situation in which a worker is using a tool (pliers) while sitting on a platform in the center of the image during a certain time period, it can be inferred that this is likely to be work process A, and if cluster Y is generated for a situation in which a worker is performing standing work on the right side of the image, it can be inferred that this is work process B. The results of clustering by analysis unit 23 are output as inference result data 27 via output unit 28, and the inference results and the basis for the inference are displayed on monitor 31 or stored in storage device 32.

[0083] The area detection unit 231 refers to the area model 24 and detects overlap between a predetermined area and the worker's skeleton. For example, by detecting overlap between a predetermined part area and the worker's skeleton, it is possible to estimate that the worker has picked up a part and started work. The posture detection unit 232 refers to each posture stored in the posture model 25 and the skeleton data 22 extracted from the video, and detects the posture of the worker in the image data. The posture model 25 stores the correspondence between each human posture and the skeleton data for that posture. By referring to the correspondence between a human posture and work, it is possible to estimate what work the worker is performing.

[0084] The background detection unit 233 detects the background from the image data by referring to a background model 291. The background model 291 stores the correspondence between each structure in the background and the appearance data of that structure. This background model 291 makes it possible to estimate the appearance data of the structure in the background, and thereby to estimate what work the worker is performing.

[0085] The object detection unit 234 detects objects from the image data using the object model 26. The object model 26 stores the correspondence between objects such as tools and their appearance data. This object model 26 makes it possible to know how tools are used and thereby detect the type of work the worker is performing. Similarly, in addition to the results detected by these detection units, the analysis unit 23 refers to the combination model 292, combines the detection results, and estimates the work content performed by the worker. The combination model 292 stores the correspondence between each work content performed by the worker and the detection results of each detection unit indicating that work content. Note that the object detection unit 234 may not only detect objects from the image data 213, but may also detect specified objects from predetermined areas, such as detecting parts from the parts removal area 901, finished products from the finished product storage area 902, and tools from the tool area 903.

[0086] The sound detection unit 235 detects the work content from the sound data 296 by referring to the sound model 293. The sound model 293 stores the correspondence between sound elements and work content. The sound data 296 makes it possible to estimate what work the worker is performing.

[0087] The movement detection unit 236 generates movement analysis data 295 by referring to the skeleton data 22, and detects the work content from this movement analysis data 295. The movement analysis model 294 stores the correspondence between the movement analysis data 295 and the work content. The movement analysis data 295 makes it possible to estimate what work the worker is performing.

[0088] Each of these configured processing units accesses the skeleton data 22 or the estimation result data 27 stored on a nonvolatile memory such as a hard disk.

[0089] The skeleton extraction unit 21 extracts skeleton data 22 of the person captured in the image data input from the video camera 1 or the video recorder 13. The analysis unit 23 receives the image data for analysis and the skeleton data for analysis 22 as input, and obtains estimation result data 27 that estimates the work to be performed by the worker. The output unit 28 outputs the estimation result data 27 to a monitor 31 and a storage device 32, which are external devices.

[0090] The object detection unit 234 uses a machine learning model to detect objects from each frame constituting the video. For the purposes of explanation, the detected object is a tool used for work, such as a screwdriver. However, the object is not limited to any particular object / pattern that appears in the image. The object detection unit 234 detects the position and type of object from image data 210 (described later) such as that shown in FIG. 31. The object detection unit 234 may detect an object not only over the entire frame, but also within a region based on a specified skeleton. For example, object detection may be limited to a range within the worker's reach and within a range that the worker can grasp. Specifically, in this embodiment, the object detection unit 234 detects a specified object, i.e., a screwdriver, within a detection range near both wrists determined from the skeleton data 22. By limiting this range, the object detection unit 234 can speed up object detection and reduce erroneous object detection.

[0091] The object detection unit 234 obtains the coordinates of the detected object and a score indicating the likelihood of the object detection, using the results of machine learning of the object model 26. The object detection unit 234 excludes detection results with clearly low scores from the results of machine learning of the object model 26, using a predetermined threshold. The object detection unit 234 also determines as a detection a result that satisfies the condition that the number of consecutive detections of an object in multiple consecutive frames is equal to or exceeds a threshold, and filters the detection results by majority voting, etc., thereby suppressing false detections and preventing detections from flickering from frame to frame.

[0092] The object detection unit 234 detects objects within a region based on a specified skeleton, such as detecting an object held by a worker. However, parts of an object are often hidden by the worker's body or work space. When a part of an object is hidden, detection accuracy decreases, making it difficult to measure work time. Therefore, when the object detection unit 234 detects, for example, a Phillips screwdriver or a flathead screwdriver, the time spent in the same working posture is estimated to be the work of tightening a screw. The position of the extracted screwdriver is superimposed on the image. The work time is then output as a graph.

[0093] In this embodiment, in a preliminary learning stage, information on the work situation at the work site is collected from the video camera 1 and microphone 12, and a plurality of clusters that reflect the work situation are generated, for example, by clustering processing. Note that the generated clusters may not only strongly reflect events specific to the work process, but may also include clusters that are not reliable enough to identify the work process. For this reason, it is advisable for the worker himself to appropriately review all the generated clusters by referring to the scores obtained during cluster generation, etc.

[0094] Then, in the actual use stage, clustering processing of six-dimensional data is performed using six sets of element information (area, posture, background, object, sound, movement) from the images and sounds obtained in the actual use stage, and a decision is made as to which existing cluster the data will be classified into, or whether a new cluster will be created. If the data is classified into cluster X as a result, it can be estimated that this situation is likely to be work process A.

[0095] Here, we have explained an example in which learning is clustering, but in essence, if it is known that the relationship between an image and sound in a work process indicates a specific work process, then it is sufficient to be able to use this relationship to determine that it is a specific work process.

[0096] According to the present invention, which takes into account the element of sound, it is expected that the estimation accuracy will be improved. For example, when only an image is used, the state of tools or equipment used in assembly or inspection may be obscured by a person's shadow, making it difficult to recognize from the image. In the above example, the accuracy is poor because the estimation is based on three-dimensional element information, which is missing the element information of the object, out of the four-dimensional element information (area, posture, background, object) obtained from the image. In this respect, in this embodiment, the element of sound is added to this, so if the snapping sound of pliers being used can be confirmed, the probability of classification into cluster X will be increased.

[0097] First, regarding sound, by learning the sounds of tools being used during work, it becomes possible to recognize hand-held tools that cannot be recognized in images because they are in the shadows, and it is possible to automatically extract with greater accuracy the situation in which a person is assembling or inspecting a product. It is best to use machine learning to estimate raw sound or voice data or data after preprocessing using FFT, etc., and it can also be determined from speech recognition.

[0098] Furthermore, by detecting the image of the object being held or used by the worker and the sound emitted from the object, it is expected that the accuracy of estimating the work content of the worker can be improved. It is best to estimate the work content using image recognition and sound recognition individually or in combination.

[0099] Furthermore, if the location where the sound or voice is generated is roughly determined, pointing a directional microphone at it will enable more accurate recognition of the sound or voice without picking up ambient noise.It is also possible to recognize the sound when a tool is taken out or put down, or when the tool is taken out, by sounding a buzzer due to a change in weight, and it is also possible to recognize the number of times an electric screwdriver has been turned from the sound.

[0100] Regarding the learning function, a score indicating the likelihood of false positives for images, skeletons, and sounds (or voice) can be obtained as a result of machine learning. The score is set to a predetermined threshold, and detection results with clearly low scores can be excluded from the results. In addition, it is a good idea to suppress false positives and prevent flickering detections by filtering the detection results based on the number of consecutive detections or majority vote.

[0101] FIG. 31 is a diagram showing an example of video input to the work management device 2. One image data 210 is generated for each frame of a moving image in which a person appears. The person shown in this image data 210 is holding an electronic device in his / her hand, but has not yet picked up a screwdriver. The parts picking area 901 is an area where a worker can pick up parts. The tool area 903 is an area where a worker can place tools used in work. The finished product storage area 902 is an area where a worker can store finished products.

[0102] Markers 911 and 912 are markers captured in image data 210 and are used to determine a person area 910 for recognizing a person by image processing unit 202. Image processing unit 202 recognizes markers 911 and 912 and sets person area 910 before executing work management processing.

[0103] In this way, the image processing unit 202 may determine, based on the markers, a person area for recognizing a person or an area for recognizing an object captured in a video. This allows the worker to recognize his or her own work area and be appropriately captured by the video camera 1. Furthermore, the work management device 2 can appropriately set a person area for recognizing a person from frames of a video captured by the video camera 1.

[0104] The image processing unit 202 may determine the person area for recognizing a person and the area for recognizing an object based on pattern matching before executing the work management process. Alternatively, the image processing unit 202 may display a screen on which the person area for recognizing a person and the area for recognizing an object can be adjusted, so that the adjustment can be performed manually.

[0105] 32 is a diagram showing an example of skeletal data 42 extracted from video input to the work management device 2, and a pre-defined parts area 901, tool area 903, and finished product storage area 902. The skeletal data 42 is the result of the skeleton extraction unit 21 extracting skeletal information of a person from image data 210, superimposed on the image data 210, and is shown by connecting each feature point with a line. The skeleton extraction unit 21 can use a known skeletal information acquisition technology such as OpenPose (URL: https: / / github.com / CMU-Perceptual-Computing-Lab / openpose).

[0106] The part taking area 901 is an area where parts are placed that is superimposed on the image data 210. When the part taking area 901 overlaps with the predetermined skeleton data 22, the analysis unit 23 infers that the worker's working status is "part taking." The tool area 903 is an area where tools are placed that is superimposed on the image data 210. When a tool is present in the tool area 903, the analysis unit 23 infers that the worker does not yet have the tool.

[0107] The finished product storage area 902 is an area where the finished product is stored, superimposed on the image data 210. When the finished product storage area 902 overlaps with the predetermined skeleton data 22, the analysis unit 23 estimates that the status of the series of work performed by the worker is "completed."

[0108] FIG. 33 is a diagram showing an example of image data 213 after synthesis. The image processing unit 202 removes unnecessary portions from the image data 210 of FIG. 32. The image processing unit 202 synthesizes the part-removing area 901, the finished product storage area 902, the tool area 903, and the person area 910 from the image data 210, and processes the image data 213 into new image data. Note that this is not limited to this, and the image processing unit 202 may perform deformation, color adjustment, filter processing, etc. on each area as needed during synthesis. By cutting out multiple areas to be recognized from the large-sized image data 210 and combining them into a single image, the image data 213 becomes smaller in size than the original image data 210.

[0109] FIG. 34 is a diagram showing an example of the skeleton data 22. The skeleton data 22 includes a number column, a feature point column, an X-axis coordinate column, a Y-axis coordinate column, and a score column. Each row stores feature points, such as a person's joint points. In the number column, a unique number is assigned to each feature point, for example, 0 for the nose, 1 for the right shoulder, and 2 for the right elbow. The feature point column stores the name of the feature point. The X-axis coordinate column stores the X-axis coordinate value of the feature point in the image data 210. The Y-axis coordinate column stores the Y-axis coordinate value of the feature point in the image data 210. The score column stores a numerical value representing the likelihood of the feature point.

[0110] FIG. 35 is a diagram showing an example of the area model 24. The area model 24 includes an area label column, a feature point number column, a judgment logic column, and a polygon coordinate column. The area label column is a column that stores the name of the area. The feature point number column is a column that stores one or more feature point numbers of the skeleton data 22. When a feature point specified in the feature point number column falls within the area specified in the polygon coordinate column, the analysis unit 23 detects that the feature point has entered the area.

[0111] The determination logic column indicates the logic for determining whether a feature point having one of these numbers has entered this region when there are multiple feature points in the feature point number column. The polygon coordinate column stores the coordinate values ​​of the polygon that represents this region.

[0112] For example, the first row of the area model 24 is recognized as the part-taking area 901 in Figure 32, and when any (logical sum) of the feature point numbers (#3 indicates the right wrist, #6 indicates the left wrist) of the worker's skeletal data 22 exists within the polygon (quadrilateral) coordinate data (four vertex coordinates), it is recognized that the worker has taken a part for the personal computer to be assembled.

[0113] Note that the "logical product" of the judgment logic indicates a judgment based on the logical product of the feature point numbers (for example, both hands), and the "logical sum" indicates a judgment based on the logical sum of the feature point numbers (for example, one hand). In other words, when one wrist of the worker enters the parts-taking area 901, an area judgment is made that "hand enters the parts-taking area."

[0114] The second row of the area model 24 is the finished product storage area 902 in Figure 32, and when any (logical OR) of the feature point numbers (#3 indicates the right wrist, #6 indicates the left wrist) of the worker's skeletal data 22 exists within the polygon (quadrilateral) coordinate data (four vertex coordinates), it is recognized that the worker has stored the assembled finished product.

[0115] The third row of the area model 24 is the tool area 903 in Figure 32, and when both (logical product) of the feature point numbers (#3 indicates the right wrist, #6 indicates the left wrist) of the worker's skeletal data 22 exist within the polygon (quadrilateral) coordinate data (four vertex coordinates), it is recognized that the worker has taken the screwdriver or put it back.

[0116] Figures 36 and 37 are diagrams showing examples of the relationship between events that occurred during the present invention and inference results. In these figures, the horizontal axis represents time progression, and the vertical axis represents, from top to bottom, the actual work process, posture information as an example of image information, sound information, and inference results. Figure 36 is a time chart showing the work content of a worker. Figure 36 illustrates a case where a section in which posture A cannot be recognized from the image is included. In this example, the actual work process was performed as work process A from time t0 to t4, and then moved on to work process B. In the image information, posture A was measured during work process A from time t0 to t2 and from time t3 to t4, but posture A was not measured between times t2 and t3. Sound information DM showed sound A during work process A throughout almost the entire period.

[0117] The inference result at this time is that, as shown on the left side of Figure 39, although there is a period T0 during which posture is not detected within the period T during which work process A is performed, there is a time period T1 during which both posture and sound are detected for at least a certain period of time within this period T, and therefore it is determined that process A is highly likely.

[0118] Fig. 37 is a time chart showing the work content of a worker. Fig. 37 shows a case including a section where it is not possible to recognize whether posture A or posture C is being performed from the image. In this example, the actual work process was work process A from time t0 to t4, after which the worker moved on to process B. In the image information, posture A or posture C was measured during work process A between times t1 and t4. In the sound information, sound A was measured during work process A throughout almost the entire period.

[0119] The inference result at this time is that, as shown on the left side of Figure 39, although there is a section in which posture A and posture C coexist within the implementation period T of work process A, there is a time period T1 within this period T in which both posture A and sound are detected for at least a certain period of time, and therefore it is determined that process A is highly likely.

[0120] Figure 38 is a time chart showing the work content of a worker. Figure 38 shows an example of determining the start and end of work from images and sounds in order to measure work time. The image shows that the start of work is determined by detecting posture A and sound A, which indicate the start of work. The image also shows that the end of work is determined by detecting posture B and sound C, which indicate the end of work.

[0121] 39 is a time chart showing the work content of the worker. Although there is a section in which posture A and posture C coexist within the period T during which work process A is performed, there is a time period T1 within this period T during which both posture A and sound are detected for at least a certain period of time, and therefore it is determined that process A is the most likely to have occurred.

[0122] 40 is a time chart showing the working posture and the work content of the worker determined by the work management device 2. The bar graph to the right of "posture" indicates that the posture of the worker detected by the posture detection unit 232 from time t1 to t4 and from time t5 to t6 is a working posture.

[0123] The bar graph to the right of "Driver Detection" indicates that the object detection unit 234 detected a driver near the worker's right or left wrist from time t2 to t3. The bar graph to the right of "Work" indicates the work content of the worker estimated by the analysis unit 23. Screw tightening is estimated as the work from time t1 to t4. This is because a driver was detected near the worker's wrist for part of time t1 to t4. In contrast, from time t5 to t6, no driver was detected, so screw tightening is not estimated.

[0124] 41A and 41B are flowcharts showing the work management process. The image processing unit 202 acquires each frame constituting a moving image as image data (step S11). The image processing unit 202 then processes multiple regions to be recognized into new image data 213 (step S12). The skeleton extraction unit 21 then extracts skeleton data 22 of the worker from the processed image data 213 (step S13).

[0125] Thereafter, the analysis unit 23 performs detection processing in parallel. Specifically, the region detection unit 231 detects a region in the frame based on the region model 24 (step S14). The posture detection unit 232 detects a posture from the skeleton data 22 based on the posture model 25 (step S15). Here, the posture detection unit 232 calculates a score (likelihood) of the detected posture. If the score is equal to or greater than a threshold, the posture is considered to have been detected, and a character string (label) indicating the detection is output.

[0126] The background detection unit 233 detects a background from the frame based on the background model 291 (step S16). Here, the background detection unit 233 calculates a score (likelihood) of the detected background. If the score is equal to or greater than a threshold, the background is considered to have been detected, and a character string (label) indicating the detection is output.

[0127] The object detection unit 234 detects an object from the frame based on the object model 26 and the skeleton data 22 (step S17). The object detection unit 234 then calculates a score (likelihood) of the detected object. If the score is equal to or greater than a threshold, the object is deemed to have been detected and a character string (label) indicating the detection is output. For object detection, feature points are defined in the object model 26, so object detection is performed by setting a range for object detection according to the coordinates of the skeleton data 22 extracted in advance. The sound detection unit 235 detects the corresponding work content based on the sound model 293 and the sound data 296 (step S18). The movement detection unit 236 detects movement based on the movement analysis model 294 and the movement analysis data 295 (step S19).

[0128] When these detection processes are completed, the process proceeds to step S20.

[0129] In step S20, the analysis unit 23 performs a filtering process based on the number of past detections to suppress flickering of the detected region, posture, background, and object. The analysis unit 23 then calculates the detection times of the region, posture, background, and object (step S21). The analysis unit 23 then estimates the task based on these calculated detection times. Specifically, the analysis unit 23 performs a process of combining these detection results (step S22).

[0130] In the process of combining the detection results, the analysis unit 23 performs calculations by combining the calculated detection results based on data defined as the combination model 292, and estimates the work content being performed by the worker.

[0131] The combination model 292 stores, for example, a combination of the logical product of the detection of the hand entering the part-removing area and the detection of the posture for removing a part, and the start of the work. Furthermore, the combination model 292 stores a combination of the logical product of the detection of the working posture and the detection of the screwdriver near the wrist, and the screw tightening work.

[0132] For example, the analysis unit 23 performs a logical AND between the detection of the hand entering the part-picking area where the parts are placed in area detection and the detection of the hand being in a position to pick up the parts in posture detection, and if the result is true, it determines that the start work is being performed.

[0133] 42 is a flowchart showing the process of associating a working posture with a driver. The analysis unit 23 determines whether the object detection unit 234 has detected the driver (step S40). If the object detection unit 234 has not detected the driver (No), the analysis unit 23 ends the process of FIG. 42. If the object detection unit 234 has detected the driver (Yes), the analysis unit 23 proceeds to step S41.

[0134] Next, the analysis unit 23 determines whether the posture detection unit 232 has detected a working posture (step S41). If the analysis unit 23 has not detected a working posture (No), the process of FIG. 42 ends, and if the analysis unit 23 has detected a working posture (Yes), the process proceeds to step S42.

[0135] In step S42, the analysis unit 23 determines that the work time is the same as the detection time of the working posture as the screw tightening work time, and ends the processing in Fig. 42. In addition to calculating the presence or absence of detection by logical operation on the detected results, the analysis unit 23 can also perform logical OR or AND on the work time.

[0136] Note that the object detected by the object detection unit 234 is not limited to a screwdriver, but may be a brush, an air gun, a soldering iron, a wrench, etc. The work content estimated by the analysis unit 23 is not limited to screw tightening, but may be painting work when a brush or air gun is detected, soldering work when a soldering iron is detected, bolt tightening work when a wrench is detected, etc.

[0137] Returning to FIG. 41B, the explanation continues. The analysis unit 23 determines whether the task state estimated from the latest frame is a task that has started (step S23). If it is a task that has started (Yes), the analysis unit 23 starts measuring the task time (step S24) and proceeds to step S27. If it is not a task that has started (No), the analysis unit 23 proceeds to step S25.

[0138] In step S25, the analysis unit 23 determines whether the task status estimated from the latest frame is a completed task. If the task status is a completed task (Yes), the analysis unit 23 ends measurement of the task time (step S26) and proceeds to step S27. If the task status is not a completed task (No), the analysis unit 23 proceeds to step S27.

[0139] In step S27, the analysis unit 23 checks the work order. For example, the analysis unit 23 may record, as a work attribute, any work that deviates from the predetermined work order. This allows the analysis unit 23 to distinguish and display work that follows the predetermined work order from work that deviates from the predetermined work order.

[0140] The analysis unit 23 then outputs the intermediate results (step S28). The analysis unit 23 then determines whether or not there are any unprocessed frames (step S29). If there are any unprocessed frames (Yes), the analysis unit 23 returns to the processing of step S11. If all frames have been processed (No), the analysis unit 23 proceeds to step S30. In step S30, the analysis unit 23 outputs the estimation result data 27 and terminates the processing of FIGS. 41A and 41B. As described in the flowcharts of FIGS. 41A and 41B, the analysis unit 23 estimates the work being performed by the worker based on the driver detection results and the worker posture detection results. This allows the details of each work to be suitably extracted even if the worker performs multiple different work tasks in similar postures.

[0141] 43 is a diagram showing a real-time screen 5 displayed by the work management device 2. The real-time screen 5 shown in FIG. 43 is a screen showing execution, interruption, and stop of analysis, as well as the results of the analysis. The output unit 28 displays this real-time screen 5 on the monitor 31. The real-time screen 5 includes an execute button 51, a pause button 52, a stop button 53, a video display area 56, a status display field 54, a status cumulative time graph 55, and a status time chart 57.

[0142] The image display area 56 is an area where the analysis results are superimposed on the input image data. The execute button 51 is used to execute the analysis. The pause button 52 is used to pause the execution of the analysis. The stop button 53 is used to stop the execution of the analysis.

[0143] The status display column 54 shows the current analysis status, displaying the start and end of work and the implementation status of main work. The status cumulative time graph 55 is a bar graph showing the cumulative time of work in the video being analyzed. The status time chart 57 shows the most recent analysis results in a time chart.

[0144] 44 is a diagram showing the history data screen 6 displayed by the work management device 2. The history data screen 6 includes time charts showing analysis results. A state time chart 60 is a time chart showing a comprehensively determined state based on the score (likelihood) of the analysis result at each time.

[0145] The "start" time chart 61 is a time chart showing a state determined based on the score of the "start" state of the subject in the video at each time. Here, the analysis unit 23 determines that the score of the "start" state is high when the subject's skeleton overlaps with the part-taking area 901 and work has not yet started.

[0146] The "take part" time chart 62 is a time chart showing a state determined based on the score of the "take part" state of the subject in the video at each time point. Here, the analysis unit 23 determines that the score of the "start" state is high when the subject's skeleton overlaps with the part-taking area 901 and the work has started.

[0147] The "driver in use" time chart 63 is a time chart showing the state determined based on the score of the "driver in use" state of the subject of the video at each time.

[0148] The "driver unused" time chart 64 is a time chart showing a state determined based on the score of the "driver unused" state of the subject of the video at each time. The "screw tightening" time chart 65 is a time chart showing a state determined based on the score of the "screw tightening" state of the subject of the video at each time.

[0149] The "right facing" time chart 66 is a time chart showing a state determined based on the score of the "right facing" state of the subject in the video at each time. The "left facing" time chart 67 is a time chart showing a state determined based on the score of the "left facing" state of the subject in the video at each time.

[0150] The "End" time chart 68 is a time chart showing a state determined based on the score of the "End" state of the subject of the video at each time. The "Placing a part" time chart 69 is a time chart showing a state determined based on the score of the "Placing a part" state of the subject of the video at each time.

[0151] 45 is a diagram showing the recognition report screen 7 displayed by the work management device 2. The recognition report screen 7 includes a legend 70, a label 71, a stacked bar graph 72, a location column 73, and a date and time column 74.

[0152] The stacked bar graph 72 indicates the cumulative time of work based on the posture recognized by the analysis unit 23. Here, the four stacked bar graphs 72 indicate the cumulative time of each work in each time slot: from 9:00 to 10:30, from 10:50 to 12:20, from 13:20 to 14:50, and from 15:20 to 16:50.

[0153] The label 71 is used to select the work to be displayed in the stacked bar graph 72. The legend 70 indicates the relationship between the display mode displayed in the stacked bar graph 72 and the work indicated by that display mode. The location column 73 indicates the location that is the subject of the recognition report screen 7. The date and time column 74 indicates the date and time that is the subject of the recognition report screen 7. Here, four stacked bar graphs indicate the cumulative work time for each day. The bar graphs can be displayed by specified unit, such as by location or by date.

[0154] According to this recognition report screen 7, the work manager can easily grasp the cumulative work time of each worker.

[0155] The configuration and effects of the present invention will be described below.

[0156] [1] A work management device (2) comprising: an image processing unit (202) that processes each frame of a video of a person into image data (213) including a person area and an object area; a skeleton extraction unit (21) that extracts skeleton data (22) of the person from the image data (213); a posture detection unit (232) that detects the posture of the person from the skeleton data (22); and an analysis unit (23) that estimates the work of the person based on the detection result of the posture of the person.

[0157] This reduces the amount of information in the image data to be processed, allowing the skeleton extraction unit (21) to process high-resolution images in real time.

[0158] [2] The work management device according to [1], wherein the skeleton extraction unit (21) extracts skeleton data (22) of a person from an area of ​​the image data (213) in which the person is to be recognized.

[0159] Even if the image data (210) of each frame of the video has high resolution, the burden of the calculation processing is reduced, and the skeleton extraction unit (21) can provide processing that is excellent in real-time processing ability.

[0160] [3] The work management device described in [1], characterized in that the image processing unit (202) cuts out an area in which a person is to be recognized and an area in which an object is to be recognized from each frame of the video, places the area in which a person is to be recognized in one and places the area in which an object is to be recognized in the other, and combines them to process the image data (213).

[0161] A person can be recognized from one of the image data (213) and an object can be recognized from the other of the image data (213). In this case, the burden of the calculation processing is small, and the system excels in real-time processing.

[0162] [4] The work management device according to [3], further comprising an object detection unit (234) that detects a predetermined object from a portion of the image data (213) in which an area in which an object is to be recognized is located.

[0163] The object detection unit (234) detects a predetermined object from a region where the predetermined object is to be recognized, so the burden of calculation processing is small and real-time processing is excellent.

[0164] [5] The work management device described in [1], characterized in that the image processing unit (202) cuts out areas in which a person is to be recognized and areas in which an object is to be recognized from each frame of the video, adds margins, and combines them to process the image data (213).

[0165] This allows the object detection unit (234) to suppress false detections in object recognition at the joints between the regions, and the skeleton extraction unit (21) to suppress false detections in skeleton extraction.

[0166] [6] The work management device described in [1], characterized in that the image processing unit (202) cuts out an area in which a person is to be recognized and an area in which an object is to be recognized from each frame of the video, resizes the cutouts, and combines them to process the image data (213).

[0167] Since each region is resized according to the object to be recognized, even if the image data (210) has high resolution, the burden of calculation processing on the object detection unit (234) and skeleton extraction unit (21) is small, and real-time processing is excellent.

[0168] [7] The work management device described in [1], characterized in that the image processing unit (202) cuts out areas in which a person is to be recognized and areas in which an object is to be recognized from each frame of the video, filters them, and combines them to process them into the image data (213).

[0169] This makes it possible to suppress false positives in object recognition and skeleton extraction.

[0170] [8] The work management device described in [1], characterized in that the image processing unit (202) adjusts the positions at which areas in which people and objects are to be recognized are cut out from each frame of the video by pattern matching.

[0171] This makes it possible to adjust for deviations in the installation position of the video camera (1) and deviations in the worker's working position.

[0172] [9] The work management device described in [1], characterized in that the image processing unit (202) adjusts the position of cutting out the area where a person is to be recognized and the area where an object is to be recognized from each frame of the video using a marker.

[0173] This makes it possible to adjust for deviations in the installation position of the video camera (1) and deviations in the worker's working position.

[0174]

[10] The work management device described in [1], characterized in that it is provided with an adjustment unit that displays a screen on which the image processing unit (202) can manually adjust the positions at which the areas in which people and objects are to be recognized are cut out from each frame of the video.

[0175] This makes it possible to adjust for deviations in the installation position of the video camera (1) and deviations in the worker's working position.

[0176]

[11] The work management device described in [1], characterized in that it comprises an output unit (28) that, based on the detection result by the posture detection unit (232), marks the feature point data of the skeleton data (22) extracted by the skeleton extraction unit (21) on the frame before processing it into the image data (213) and displays it on a display unit.

[0177] This allows the feature point data of the skeleton data (22) to be displayed on the image of the work place captured by the video camera (1).

[0178]

[12] The work management device described in [1] further comprises a storage unit for a posture model in which a posture label is associated with each skeletal data, and the posture detection unit (232) detects the posture of the person from the skeletal data of the person based on the posture model.

[0179] This allows the posture detection unit (232) to appropriately detect the specific posture corresponding to the work of the worker.

[0180]

[13] The work management device described in claim 1, characterized in that the analysis unit (23) further estimates the person's work by using, alone or in combination, the detection results of the area, background, object, sound, movement, and posture of the person.

[0181] This makes it possible to more accurately estimate the person's activity based on a combination of posture detection results and other factors.

[0182]

[14] A work management system comprising: an image processing unit (202) that processes each frame of a video of a person into image data (213) including an area where the person is to be recognized and an area where an object is to be recognized; a skeleton extraction unit (21) that extracts skeleton data (22) of the person from the image data (213); a posture detection unit (232) that detects the posture of the person from the skeleton data (22); and an analysis unit (23) that estimates the work of the person based on the detection result of the posture of the person.

[0183] This reduces the amount of information in the image data to be processed, allowing the skeleton extraction unit to process high-resolution images in real time.

[0184]

[15] A work management method comprising the steps of: an image processing unit (202) processing each frame of a video of a person into image data (213) including an area where the person is to be recognized and an area where an object is to be recognized; a skeleton extraction unit (21) extracting skeleton data (22) of the person from the image data (213); a posture detection unit (232) detecting the posture of the person from the skeleton data (22); and an analysis unit (23) estimating the work of the person based on the detection result of the posture of the person.

[0185] This reduces the amount of information in the image data to be processed, allowing the skeleton extraction unit to process high-resolution images in real time.

[0186] <<Modifications>> The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. It is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0187] The above-described configurations, functions, processing units, processing means, etc. may be implemented in part or in whole by hardware such as an integrated circuit. The above-described configurations, functions, etc. may also be implemented by software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a recording medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0188] In each embodiment, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected.

[0189] REFERENCE SIGNS LIST 1 Video camera 13 Video recorder 14 Input / output device 2 Work management device 21 Skeleton extraction unit 22 Skeleton data 211 Model generation unit 202 Image processing unit 213 Image data 231a Area definition unit 232a Posture learning unit 233a Background definition unit 237a Procedure learning unit 23 Analysis unit 231 Area detection unit 232 Posture detection unit 233 Background detection unit 234 Object detection unit 235 Sound detection unit 236 Movement detection unit 237 Procedure detection unit 24 Area model 25 Posture model 26 Object model 22 Skeleton data 27 Estimation result data 28 Output unit 31 Monitor 32 Storage device 33 Application device 901 Parts removal area 903 Tool area 902 Finished product storage area 910 Person area

Claims

1. A work management device comprising: an image processing unit that processes each frame of a video of a person into image data including an area in which the person is to be recognized and an area in which an object is to be recognized; a skeleton extraction unit that extracts skeleton data of the person from the image data; a posture detection unit that detects the posture of the person from the skeleton data; and an analysis unit that estimates the work of the person based on the detection result of the posture of the person.

2. The work management device according to claim 1, characterized in that the skeleton extraction unit extracts skeleton data of a person from an area of ​​the image data in which a person is desired to be recognized.

3. The work management device of claim 1, characterized in that the image processing unit cuts out areas in which people are to be recognized and areas in which objects are to be recognized from each frame of the video, places the areas in which people are to be recognized in one and the areas in which objects are to be recognized in the other, and combines them to process the image data.

4. The work management device according to claim 3, further comprising an object detection unit that detects a predetermined object from a portion of the image data in which an area in which an object is to be recognized is located.

5. The work management device according to claim 1, characterized in that the image processing unit cuts out areas in which people are to be recognized and areas in which objects are to be recognized from each frame of the video, adds margins and combines them to process the image data.

6. The work management device according to claim 1, characterized in that the image processing unit cuts out areas in which people are to be recognized and areas in which objects are to be recognized from each frame of the video, resizes them, and combines them to process the image data.

7. The work management device according to claim 1, characterized in that the image processing unit cuts out areas in which people are to be recognized and areas in which objects are to be recognized from each frame of the video, filters them, and combines them to process them into image data.

8. The work management device according to claim 1, characterized in that the image processing unit adjusts the positions for cutting out areas in which people and objects are to be recognized from each frame of the video by pattern matching.

9. The work management device according to claim 1, characterized in that the image processing unit adjusts the positions for cutting out areas in which people and objects are to be recognized from each frame of the video using markers.

10. The work management device according to claim 1, further comprising an adjustment unit that displays a screen on which the image processing unit can manually adjust the positions at which it cuts out areas in which it is desired to recognize people and areas in which it is desired to recognize objects from each frame of the video.

11. The work management device according to claim 1, further comprising an output unit which, based on the results of detection by the posture detection unit, marks the feature point data of the skeletal data extracted by the skeleton extraction unit on the frame before it is processed into the image data, and displays the marking on a display unit.

12. The work management device according to claim 1, further comprising a memory unit for a posture model in which a posture label is associated with each skeletal data, and the posture detection unit detects the posture of the person from the skeletal data of the person based on the posture model.

13. The work management device according to claim 1, characterized in that the analysis unit further estimates the work of the person by using, alone or in combination, the detection results of the area, background, object, sound, movement, and posture of the person.

14. A work management system comprising: an image processing unit that processes each frame of a video of a person into image data including an area in which the person is desired to be recognized and an area in which an object is desired to be recognized; a skeleton extraction unit that extracts skeleton data of the person from the image data; a posture detection unit that detects the posture of the person from the skeleton data; and an analysis unit that estimates the work of the person based on the detection result of the person's posture.

15. A work management method comprising the steps of: an image processing unit processing each frame of a video of a person into image data including an area in which a person is desired to be recognized and an area in which an object is desired to be recognized; a skeleton extraction unit extracting skeleton data of the person from the image data; a posture detection unit detecting the posture of the person from the skeletal data; and an analysis unit estimating the work of the person based on the detection result of the posture of the person.

Images