Work analysis device

By generating object detection models and joint position operation estimation models, and combining them with operation label assignment and automatic adjustment of judgment criteria, the problems of low visualization of operator operation data and low judgment accuracy are solved, and efficient and accurate operation judgment is achieved.

CN118985003APending Publication Date: 2024-11-19FANUC LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093997.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In existing technologies, it is difficult to visualize the operator's work data, and the existing models are complex and have low interpretability. Work judgment requires a lot of calculation and manual annotation, and the judgment accuracy is not easy to improve.

Method used

By generating object detection models and joint position operation estimation models, and combining them with operation label assignment and automatic adjustment of judgment criteria, accurate judgment of operators' operations can be achieved.

Benefits of technology

It enables automatic adjustment of judgment criteria, improves the accuracy and efficiency of job judgment, reduces the workload of manual annotation, and improves the accuracy of job judgment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118985003A_ABST
    Figure CN118985003A_ABST
Patent Text Reader

Abstract

In order to accurately determine work, a determination reference, i.e., a parameter, is automatically adjusted and obtained. A work analysis device is provided with: a work tag application unit that applies a work tag indicating the work by a worker to video data including the work by the worker; an object detection annotation unit for annotating an object associated with the work by the operator to the video data to which the work tag has been assigned; an object detection learning unit that generates an object detection model for performing object detection from the image data of the object annotated by the object detection annotation unit; an object detection unit that detects an object from the image data using the object detection model; a job determination parameter calculation unit that performs job determination of the video data to which the job tag has been assigned, and calculates a determination reference that minimizes an error with the assigned job tag; and a work determination unit that uses the object detection model and the determination reference to determine the work of the operator in the newly input video data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a work analysis device. Background Art

[0002] In factories, it is possible to obtain the operating data of machine tools, etc., but it is impossible to obtain the data of operators' work. Therefore, in order to achieve work improvement, robot introduction research, factory digital twin, etc., it is necessary to visualize the operator's work, and it is important to have technology that can automatically recognize what the operator is doing from the image of the operator's work.

[0003] In this regard, the following technology is known: machine learning is performed using learning object data consisting of input data of an image obtained by photographing a worker's work and label data of the worker's work represented by the image, a learned model for identifying the work from the image is generated, and the learned model is used to identify which work is being performed in the image of the analysis object. For example, refer to Patent Document 1.

[0004] In addition, there is known a technique for determining the position of a worker's hand based on image data with depth captured by a depth sensor, and determining the position of an object based on image data captured by a digital camera, thereby determining the content of the action performed by the worker during the operation. For example, see Patent Document 2.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2021-67981

[0008] Patent Document 2: International Publication No. 2017 / 222070 Summary of the invention

[0009] Problems to be solved by the invention

[0010] However, a classification model such as the learned model of Patent Document 1 has problems of being complex and having low interpretability.

[0011] In addition, as in Patent Document 2, a tool (object) being used is detected from within an image in order to classify the work, and thus a large amount of calculation is required to scan the entire image.

[0012] Furthermore, in order to accurately determine the work being performed by the operator, it is necessary to adjust the determination criteria (parameters) for work determination and manually find images of various work scenes for annotation, which takes time. In addition, there is a problem that even if the images are found manually, it is still unknown whether the accuracy of work determination has been improved.

[0013] Therefore, in order to accurately determine the work, a function of automatically adjusting and obtaining a determination criterion (parameter) is desired.

[0014] Means for solving problems

[0015] One embodiment of the job analysis device disclosed herein is a job analysis device for analyzing a worker's job, comprising: a job label assignment unit that assigns a job label representing the worker's job to image data containing the worker's job; an object detection annotation unit that annotates the image data assigned the job label with an object associated with the worker's job; an object detection learning unit that generates an object detection model for performing object detection based on the image data of the object annotated by the object detection annotation unit; an object detection unit that detects the object from the image data using the object detection model; a job judgment parameter calculation unit that performs job judgment on the image data assigned the job label and calculates a judgment criterion that minimizes the error with the assigned job label; and a job judgment unit that uses the object detection model and the judgment criterion to judge the worker's job in newly input image data.

[0016] Effects of the Invention

[0017] According to one embodiment, in order to accurately determine the work, the determination criterion (parameter) can be automatically adjusted and obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a functional block diagram showing an example of the functional configuration of the work analysis system according to the first embodiment.

[0019] Figure 2 This is a diagram showing an example of a work table.

[0020] Figure 3 This is a diagram showing an example of a user interface for assigning a job label.

[0021] Figure 4 This is a diagram showing an example of image data showing different states of a grinding machine.

[0022] Figure 5 This is a diagram showing an example of the determination result of the work determination.

[0023] Figure 6 This is a diagram showing an example of misdetection.

[0024] Figure 7 This is a diagram showing an example of an image region in video data.

[0025] Figure 8 It is a diagram showing an example of the operation of the moving object detection unit.

[0026] Fig. 9 This is a flowchart for explaining parameter calculation processing of the work analysis device.

[0027] Fig.10 This is a flowchart for explaining the analysis process of the work analysis device.

[0028] Fig.11 This is a functional block diagram showing an example of the functional configuration of the work analysis system according to the second embodiment.

[0029] Fig.12 This is a diagram showing an example of joint position information in a frame image.

[0030] Fig.13 This is a diagram showing an example of the operation of the joint position work estimation model.

[0031] Fig.14 This is a flowchart for explaining parameter calculation processing of the work analysis device.

[0032] Fig.15 This is a flowchart for explaining the analysis process of the work analysis device. DETAILED DESCRIPTION

[0033] A first embodiment and a second embodiment of the work analysis device will be described in detail with reference to the drawings.

[0034] Here, the various embodiments are common in the following structural aspects: a job label representing the worker's job is assigned to the image data (dynamic image) obtained by pre-shooting the worker's work, the worker annotates the image data assigned the job label with objects (tools) associated with the job, and an object detection model for object detection is generated based on the image data of the annotated object.

[0035] However, in determining the work of the operator, in the first embodiment, the generated object detection model is used to perform work determination of the work of the operator in the image data to which the work label is assigned, and a determination criterion that minimizes the error with the assigned work label is calculated, thereby determining the work of the operator in the newly input image data using the object detection model and the calculated determination criterion. In contrast, in the second embodiment, the difference from the first embodiment is that joint position information related to the joints of the operator is estimated, a joint position work estimation model for estimating the work of the operator is generated based on the estimated joint position information and the assigned work label, and a determination criterion is calculated based on a value related to the accuracy of object detection in work determination using the object detection model and a classification probability of the work estimated based on the joint position in work determination using the joint position work estimation model so as to minimize the error with the work label, and the object detection model, the joint position work estimation model, and the determination criterion are used to determine the work of the operator in the newly input image data.

[0036] Hereinafter, the first embodiment will be described in detail first, and then the second embodiment will be described mainly focusing on the parts different from the first embodiment.

[0037] <First Embodiment>

[0038] Figure 1 This is a functional block diagram showing an example of the functional configuration of the work analysis system according to the first embodiment.

[0039] like Figure 1 As shown, the work analysis system 100 includes a work analysis device 1 and a camera 2 .

[0040] The work analysis device 1 and the camera 2 can be connected to each other via a network (not shown) such as a LAN (Local Area Network) or the Internet. In this case, the work analysis device 1 and the camera 2 are provided with a communication unit (not shown) for communicating with each other via the above connection. In addition, the work analysis device 1 and the camera 2 can also be directly connected to each other by wire or wirelessly via a connection interface (not shown).

[0041] In addition, Figure 1 In the embodiment, the work analysis device 1 is connected to one camera 2, but may be connected to two or more cameras 2.

[0042] The camera 2 is a digital camera or the like, and captures a two-dimensional frame image at a predetermined frame rate (e.g., 30 fps) by projecting objects such as workers and tools (not shown) onto a plane perpendicular to the optical axis of the camera 2. The camera 2 outputs the captured frame image as image data to the work analysis device 1. In addition, the image data captured by the camera 2 may be a visible light image such as an RGB color image, a grayscale image, or a depth image.

[0043] <Work Analysis Device 1>

[0044] The operation analysis device 1 is a computer known to those skilled in the art, such as Figure 1 As shown, the control unit 10 and the storage unit 20 are provided. In addition, the control unit 10 includes a job registration unit 101, a job label assignment unit 102, an object detection annotation unit 103, an object detection learning unit 104, a job determination parameter calculation unit 105, an object detection annotation suggestion unit 106, and a job determination unit 107. In addition, the job determination unit 107 includes an object detection unit 1071 and a moving body detection unit 1072.

[0045] The storage unit 20 is a storage device such as a ROM (Read Only Memory) or a HDD (Hard Disk Drive). The storage unit 20 stores an operating system and application programs executed by the control unit 10 described later. The storage unit 20 includes an image data storage unit 201, a job registration storage unit 202, and an input data storage unit 203.

[0046] The image data storage unit 201 stores image data of objects such as workers and tools captured by the camera 2 .

[0047] In the job registration storage unit 202, for example, a job table pre-registered by the job registration unit 101 described later is stored based on input operations performed by users such as workers via input devices such as keyboards and touch panels (not shown) included in the job analysis device 1. The job table associates the tool (object) detected by the object detection unit 1071 described later with the corresponding worker's job.

[0048] Figure 2 This is a diagram showing an example of a work table.

[0049] like Figure 2 As shown, the work table has storage areas for "objects" and "work".

[0050] In the storage area of ​​"object" in the work table, tool names such as "leutor (registered trademark)" and "sandpaper" are stored, for example.

[0051] In the storage area of ​​"job" in the job table, jobs such as "grinding" and "rubbing" are stored, for example.

[0052] In the input data storage unit 203, as input data for the object detection learning unit 104 described later to generate an object detection model, for example, a collection of frame image data is stored that associates a tool (object) annotated by the object detection annotation unit 103 described later in a frame image of image data with an image range in which the tool is photographed.

[0053] The control unit 10 includes a CPU, a ROM, a RAM (Random Access Memory), a CMOS memory, and the like, and these are configured to be able to communicate with each other via a bus, which is well known to those skilled in the art.

[0054] The CPU is a processor that controls the entire work analysis device 1. The CPU reads out the system program and application program stored in the ROM via the bus, and controls the entire work analysis device 1 according to the system program and application program. Figure 1 As shown, the control unit 10 is configured to realize the functions of a job registration unit 101, a job label assignment unit 102, an object detection annotation unit 103, an object detection learning unit 104, a job determination parameter calculation unit 105, an object detection annotation suggestion unit 106, and a job determination unit 107. In addition, the job determination unit 107 is configured to realize the functions of an object detection unit 1071 and a moving body detection unit 1072. Various data such as temporary calculation data and display data are stored in the RAM. The CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory that maintains the storage state even if the power supply of the job analysis device 1 is turned off.

[0055] The job registration unit 101 registers the relationship between the tool (detected object) used and the job (job to be identified) using the tool (object) in correspondence with each other based on an input operation performed by a user such as an operator via an input device (not shown) of the job analysis device 1. Figure 2 in the worksheet shown.

[0056] For example, the job label assigning unit 102 assigns a job label indicating the job name of the worker to the image data (moving image data) containing the worker's job, which is displayed by the user and stored in the image data storage unit 201 .

[0057] Figure 3 An example of the user interface 30 for assigning a job label is shown.

[0058] like Figure 3As shown, the user interface 30 has: an area 301 for reproducing image data (dynamic images) stored in the image data storage unit 201, a reproduction stop button 302, a slider 303, an area 310 for representing the job labels assigned to the image data by the job label assigning unit 102 in a time series, a grinder button 321 for representing the tool annotated by the object detection annotation unit 103 described later, a micro grinder button 322, a sandpaper button 323, a wiping cloth button 314, and a completion button 330 for completing the assignment of job labels and / or annotation of objects.

[0059] Specifically, the job label assigning unit 102 displays the user interface 30 on a display device (not shown) such as an LCD included in the job analysis device 1, and reproduces the image data (dynamic image data) stored in the image data storage unit 201 in the area 301 of the user interface 30. The user operates the reproduction stop button 302 and the slider 303 via the input device (not shown) of the job analysis device 1 to confirm the image data. When the user confirms the operation of the operator "grinding" in the image data from time 13:10 to time 13:13, the user inputs the job name "grinding", and the job label assigning unit 102 assigns the job label "grinding" to the image data from time 13:10 to time 13:13. In addition, when the user confirms the operation of the operator "fine grinding" in the image data from time 13:13 to time 13:18, the user inputs the job name "fine grinding", and the job label assigning unit 102 assigns the job label "fine grinding" to the image data from time 13:13 to time 13:18. In addition, when the user confirms the operation of the operator "rubbing" in the image data from time 13:18 to time 13:20, the user inputs the operation name of "rubbing", and the operation label assignment unit 102 assigns the operation label of "rubbing" to the image data from time 13:18 to time 13:20. In addition, when the user confirms the operation of the operator "cleaning" in the image data from time 13:20 to time 13:22, the user inputs the operation name of "cleaning", and the operation label assignment unit 102 assigns the operation label of "cleaning" to the image data from time 13:20 to time 13:22.

[0060] The task labeling unit 102 may display the task labeling result in time series on a display device (not shown) of the task analysis device 1 in the area 310. Then, the task labeling unit 102 outputs the image data to which the task label is added to the object detection annotation unit 103.

[0061] The object detection and annotation unit 103 annotates, for example, a tool (object) associated with the work of the worker to the image data to which the work label is given.

[0062] Specifically, the object detection annotation unit 103 displays, for example, in area 301 of the user interface 30, frame images (still images) divided at predetermined intervals of a tool (object) of a grinder captured in image data assigned a job label of "grinding" from time 13:10 to time 13:13, or frame images (still images) divided at arbitrary intervals by the user.

[0063] In addition, it is preferable to set a predetermined interval or an arbitrary interval so that, for example, about 20 frame images (still images) are displayed for each job ticket.

[0064] This eliminates the need for the user to check video data for several hours, and enables the user to perform work efficiently, thereby reducing the burden on the user.

[0065] The object detection annotation unit 103 is based on the user's input operation, such as Figure 3 As shown, the image range (bold rectangle) of the tool (object) captured in each frame image (still image) is obtained, and the tool (object) is annotated as a grinder by pressing the grinder button 321, etc. In addition, the object detection annotation unit 103 also obtains the image range of the tool (object) captured in the frame image (still image) in which the tool (object) is captured, and annotates the tool (object) in the same manner as in the case of "grinding", for each image data to which the operation labels "fine grinding", "rubbing", and "cleaning" are assigned, and annotates the tool (object).

[0066] When the object detection annotation unit 103 completes the acquisition of the image range of the tool (object) and the annotation of the tool (object) for all frame images (still images) of the image data assigned a job label, and the user presses the completion button 330, a collection of frame image data (hereinafter also referred to as "annotated frame image data") is stored in the input data storage unit 203, which corresponds the image range of the frame image (still image) showing the tool (assigned with a timestamp) in the image data (dynamic image data) of the time when each job is performed (the time from the start of the job to the end of the job) and the annotated tool (object).

[0067] The object detection learning unit 104 generates an object detection model for detecting an object based on the image data of the annotated object.

[0068] Specifically, the object detection learning unit 104 performs well-known machine learning using training data, for example, to generate a learning model such as a neural network, that is, an object detection model, wherein the training data uses the annotated frame image data stored in the input data storage unit 203 as input data and the annotated tool (object) as label data. The object detection learning unit 104 stores the generated object detection model in the storage unit 20.

[0069] The work determination parameter calculation unit 105 uses the object detection model generated by the object detection learning unit 104 to perform work determination on the image data to which the work label is assigned, and calculates a determination criterion that minimizes the error with the assigned work label.

[0070] Specifically, the work determination parameter calculation unit 105 calculates, for example, Figure 2 For each job registered in the job table, an initial value of a parameter serving as a judgment standard is set. In addition, the parameters include, for example, the number of seconds X that the job is performed after the object is detected, the threshold value of a value related to the accuracy of object detection for judging that the job "grinding" is being performed, and the threshold value of a value related to the accuracy of object detection for judging that the job "friction" is being performed. Since the number of seconds X that the job is performed after the object is detected is included in the parameter, the job analysis device 1 can judge that a job using the tool is being performed, even if the tool (object) cannot be detected by the image data, as long as the tool (object) is detected within the last X seconds.

[0071] The operation determination parameter calculation unit 105 inputs the annotated frame image data among other image data to which the operation label is assigned stored in the input data storage unit 203 into the object detection model to detect the tool (object). Figure 2 The job table is used to judge the job, and the error between the judged job and the correct job label is calculated. Then, the job judgment parameter calculation unit 105 calculates the evaluation index such as the F1 score of the parameter value for each job based on the error calculated from all the annotated frame image data, and calculates the parameter value of each job by Bayesian optimization or the like in a manner that maximizes the calculated evaluation index of each job.

[0072] The object detection annotation suggestion unit 106 performs job determination on the image data to which the job label is assigned using the parameter (determination criterion) calculated by the job determination parameter calculation unit 105 , and suggests a frame image (still image) to be annotated based on the determination result of the job determination.

[0073] For example, in object detection in a grinding machine, Figure 4 As shown in the upper part of the figure, when learning is performed only based on the image data of the worker holding the hand, Figure 4 As described in the next paragraph of , the value related to the accuracy of object detection of a grinder placed on a workbench or the like is reduced. Therefore, image data annotated in a wide variety of scenes is required, but it is troublesome for users to find various scenes. Therefore, as described later, the object detection annotation suggestion unit 106 suggests frame images (still images) that are preferably automatically annotated.

[0074] Specifically, the object detection annotation suggestion unit 106 performs task determination using, for example, image data that associates a tool (object) annotated in other image data to which a task label is assigned and stored in the input data storage unit 203 with an image range in which the tool is captured.

[0075] Figure 5 This is a diagram showing an example of the determination result of the work determination. Figure 5 The upper part of represents the time series of the correct job labels assigned to the other image data. Figure 5 The middle section of shows the determination result of the operator's work made by the object detection annotation suggestion unit 106 on the image data using the object detection model and parameters. Figure 5 The lower part of represents the object detection result in the image data based on the object detection model.

[0076] like Figure 5 As shown, there is a time period from 13:40 to 13:43 when the correct job label is "grinding" and "grinding" is not determined (detected) in the job determination result. This is because, for example, during the period from X seconds after the object is detected to the number of seconds X when the job is performed, the frame image (still image) in which the grinder is captured is not extracted regardless of whether there is a frame image (still image) in which the grinder is captured, in order to increase the value related to the accuracy of object detection. Therefore, in order to increase the value related to the accuracy of object detection, the object detection annotation suggestion unit 106 extracts the frame image (still image) in which the grinder is captured from the other image data at the time period in which the job determination result is not determined (detected) as "grinding". The object detection annotation suggestion unit 106 displays the extracted frame image (still image) on the user interface 30, obtains the image range of the grinder in the extracted frame image (still image) based on the user's input operation, and annotates it as a grinder by pressing the grinder button 321. The object detection annotation suggestion unit 106 stores, in the input data storage unit 203 , image data in which an image range of a frame image (still image) (with a time stamp) showing a grinder is associated with the annotated grinder.

[0077] In addition, if Figure 5 As shown, from 13:43 to 13:50 when the correct operation label is "rubbing", there are times when "grinding" is erroneously determined (erroneously detected) in the operation determination result and times when "rubbing" is not determined (detected). The reason for the erroneous determination (erroneous detection) of "grinding" is that a grinder is erroneously detected in the object detection of the frame image (still image) at 13:43. In addition, the reason why "rubbing" is not determined (detected) is that during the period from X seconds after the object is detected in the parameter until the number of seconds X of the operation, no matter whether there is a frame image (still image) in which sandpaper is captured, it is not extracted.

[0078] Therefore, in order to increase the value related to the accuracy of object detection, the object detection annotation suggestion unit 106 extracts the frame image (still image) in which the sandpaper is photographed around the time 13:43 and the frame image (still image) in which the sandpaper is photographed at a time when it is not determined (detected) as "friction" from the other image data. The object detection annotation suggestion unit 106 displays the extracted frame images (still images) on the user interface 30, obtains the image range of the sandpaper in each frame image (still image) based on the user's input operation, and annotates the tool (object) as sandpaper by pressing the sandpaper button 323. The object detection annotation suggestion unit 106 stores image data in which the image range of the frame image (still image) showing the sandpaper (with a time stamp) is associated with the annotated sandpaper in the input data storage unit 203.

[0079] This makes it possible to improve the accuracy of object detection without requiring the user to spend time searching for various scenes.

[0080] Furthermore, the object detection annotation suggestion unit 106 may also extract a frame image (still image) in which the tool (object) is photographed when the value related to the accuracy of object detection, i.e., the reliability of object detection of the tool (object), is lower than a predetermined value (e.g., 20%). The object detection annotation suggestion unit 106 may also display the extracted frame image (still image) on the user interface 30, obtain the image range of the tool (object) in the extracted frame image (still image), and annotate the tool (object) based on the user's input operation.

[0081] After that, the object detection learning unit 104 performs machine learning using the following image data, wherein the image data includes a frame image (still image) annotated with the tool (object) extracted (suggested) by the object detection annotation suggestion unit 106, and updates the object detection model. The operation determination parameter calculation unit 105 determines the operation by inputting the annotated frame image data including the frame image (still image) extracted (suggested) by the object detection annotation suggestion unit 106 into the updated object detection model, and calculates the error between the assigned correct operation label and the operation determination result. The operation determination parameter calculation unit 105 calculates the evaluation index such as the F1 score of the parameter value for each operation based on the calculated error, and recalculates the parameter value of each operation by Bayesian optimization or the like so that the calculated evaluation index of each operation becomes the maximum. For example, the object detection learning unit 104 and the operation determination parameter calculation unit 105 repeat the process until the frame image (still image) extracted (suggested) by the object detection annotation suggestion unit 106 disappears or is less than a predetermined number. Then, the object detection learning unit 104 outputs the generated object detection model to the object detection unit 1071 described later, and the operation determination parameter calculation unit 105 outputs the calculated parameter to the operation determination unit 107 described later.

[0082] The work determination unit 107 determines the work of the worker in the video data newly input from the camera 2 using the object detection model and the set parameters (determination criteria).

[0083] Specifically, the work determination unit 107 inputs, for example, a frame image (still image) of video data newly input from the camera 2 to an object detection model of an object detection unit 1071 described later and a moving body detection unit 1072 described later. The work determination unit 107 determines the tool (object) based on the detection result of the tool (object) output from the object detection model, the detection result of the moving body detection unit 1072, Figure 2 In addition, when the tool (object) cannot be detected by the frame image (still image) of the image data, the operation determination unit 107 can determine the operation of the operator in the frame image based on the parameter of the number of seconds X of the operation performed within X seconds after the object is detected, when the tool (object) is detected at the nearest position within X seconds of the frame image.

[0084] In addition, the work determination unit 107 may determine that the worker's work is "no work" when the value related to the accuracy of object detection, such as the reliability of object detection or the classification probability of the level output from the object detection model of the object detection unit 1071, is below a preset threshold value (e.g., 70%). Figure 6As shown, when the worker only touches the workpiece in the image data, the work judgment unit 107 can, upon receiving object detection results such as "sandpaper" and "reliability 40%", judge the worker's work as "no work" because the reliability is below the threshold (for example, 70%, etc.).

[0085] This can reduce erroneous detection of work.

[0086] The object detection unit 1071 has an object detection model generated by the object detection learning unit 104, inputs a frame image (still image) of image data newly input from the camera 2 into the object detection model, and outputs values ​​related to the accuracy of object detection such as the detection result of the tool (object) and reliability.

[0087] The moving object detection unit 1072 detects a moving object such as a worker or a tool based on changes such as changes in brightness of pixels in a designated image region in each frame image (still image) of video data newly input from the camera 2 .

[0088] Specifically, if Figure 7 As shown, if there is movement such as a change in pixel brightness in the image region indicated by a thick rectangle in the frame image (still image), the moving object detection unit 1072 can determine that the worker in the image data is performing work.

[0089] In addition, if Figure 8 As shown in the upper part of FIG. 1 , when the moving body detection unit 1072 detects movement at regular intervals within X seconds (e.g., 5 seconds, etc.) indicated by the dotted rectangle, it can also be determined that the operator is continuously performing the work. Figure 8 As shown in the lower part of , the moving body detection unit 1072 can also determine that work is being performed using the detected tool (object) during the period when the activity of the moving body is detected, when a tool (object) such as a grinder is detected by the object detection unit 1071 from the frame image (still image) at the moment represented by the shaded rectangle.

[0090] On the other hand, the moving object detection unit 1072 may determine that the worker is not working when no movement is detected for more than X seconds.

[0091] <Parameter Calculation Processing of Work Analysis Device 1>

[0092] Next, the operation of the parameter calculation process of the work analysis device 1 according to the first embodiment will be described.

[0093] Fig. 9This is a flowchart for explaining the parameter calculation process of the work analysis device 1. The flow shown here is executed when a user such as a worker registers a new tool (object) and work in the work table, for example.

[0094] In step S1 , the work label assigning unit 102 reproduces the image data including the work of the worker stored in the image data storage unit 201 on the user interface 30 , and assigns a work label indicating the work being performed by the worker to the image data according to the user input operation.

[0095] In step S2, the object detection and annotation unit 103 obtains the image range of the tool (object) captured for the frame images (still images) divided at predetermined intervals for each job label in the image data to which the job label is assigned in step S1, and annotates the tool (object). The object detection and annotation unit 103 stores the annotated frame image data in the input data storage unit 203, which associates the image range of the frame image (still image) (with a time stamp) showing the tool in the image data (dynamic image data) at the time when each job is performed (from the start of the job to the end of the job) with the annotated tool (object).

[0096] In step S3 , the object detection learning unit 104 generates an object detection model for performing object detection based on the annotated frame image data annotated in step S2 .

[0097] In step S4 , the work determination parameter calculation unit 105 inputs the annotated frame image data of other image data to which the work label is assigned and stored in the input data storage unit 203 into the object detection model to detect the tool (object).

[0098] In step S5 , the work determination parameter calculation unit 105 determines the work of the worker based on the object detection result in step S4 and the work table.

[0099] In step S6 , the job determination parameter calculation unit 105 calculates the error between the correct job label and the determination result in step S5 for each job.

[0100] In step S7 , based on the errors calculated in all the image data, an evaluation index such as an F1 score of the parameter value is calculated for each job.

[0101] In step S8 , the work determination parameter calculation unit 105 calculates the parameters of each work by Bayesian optimization or the like so as to maximize the evaluation index of each work.

[0102] In step S9 , the object detection and annotation suggestion unit 106 uses the parameter (determination criterion) calculated in step S8 to perform job determination on other image data to which the job label is assigned.

[0103] In step S10, the object detection annotation suggestion unit 106 determines whether there is a frame image (still image) recommended for increasing the value related to the accuracy of object detection in a portion where the value related to the accuracy of object detection is low, such as false detection or non-detection, based on the determination result of step S9. If there is a recommended frame image (still image), the process returns to step S2, and the process from step S2 to step S9 is performed again, including the recommended frame image (still image). On the other hand, if there is no recommended frame image (still image), the work analysis device 1 sets the object detection model generated in step S3 in the object detection unit 1071, and sets the parameters calculated in step S8 in the work determination unit 107, and ends the parameter calculation process.

[0104] <Analysis Processing of Work Analysis Device 1>

[0105] Next, the operation of the analysis process of the work analysis device 1 according to the first embodiment will be described.

[0106] Fig.10 This is a flowchart for explaining the analysis process of the work analysis device 1. The flow shown here is repeatedly executed while the video data is input from the camera 2.

[0107] In step S21 , the object detection unit 1071 inputs a frame image (still image) of video data newly input from the camera 2 into the object detection model to detect a tool (object).

[0108] In step S22 , the moving object detection unit 1072 detects a moving object such as a worker or a tool based on changes such as changes in brightness of pixels in a designated image region of each frame image (still image) of video data newly input from the camera 2 .

[0109] In step S23 , the work determination unit 107 determines the work of the worker based on the detection result of the tool (object) in step S21 , the detection result of the moving body in step S22 , the set parameters, and the work table.

[0110] As described above, the work analysis device 1 of the first embodiment can automatically adjust the determination criteria in order to accurately determine the work. In other words, the user only needs to label the work and annotate the object to automatically calculate the optimal parameters.

[0111] Furthermore, when the accuracy of work determination is insufficient, the work analysis device 1 can automatically suggest frames in a moving image that can improve the accuracy of work determination if annotation is performed.

[0112] The first embodiment has been described above.

[0113] <Second Embodiment>

[0114] Next, the second embodiment is described. In the first embodiment, the generated object detection model is used to perform a work judgment of the work of the operator in the image data to which the work label is assigned, and a judgment criterion that minimizes the error with the assigned work label is calculated, thereby using the object detection model and the calculated judgment criterion to judge the work of the operator in the newly input image data. In contrast, in the second embodiment, the difference from the first embodiment is that joint position information related to the joints of the operator is estimated, and a joint position work estimation model that estimates the work of the operator is generated based on the estimated joint position information and the assigned work label. Based on the value related to the accuracy of object detection in the work judgment using the object detection model and the classification probability of the work estimated according to the joint position in the work judgment using the joint position work estimation model, the judgment criterion is calculated to minimize the error with the work label, and the object detection model, the joint position work estimation model, and the judgment criterion are used to judge the work of the operator in the newly input image data.

[0115] Thus, the work analysis device 1A according to the second embodiment can automatically adjust the determination criterion in order to accurately determine the work.

[0116] Hereinafter, a second embodiment will be described.

[0117] Fig.11 is a functional block diagram showing an example of a functional structure of a work analysis system according to a second embodiment. Figure 1 Elements having the same functions as the elements of the work analysis system 100 are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0118] like Fig.11 As shown, the work analysis system 100 includes a work analysis device 1A and a camera 2 .

[0119] The camera 2 has the same function as that of the camera 2 in the first embodiment.

[0120] <Work Analysis Device 1A>

[0121] like Fig.11As shown, the work analysis device 1A includes a control unit 10a and a storage unit 20. In addition, the control unit 10a has a work registration unit 101, a work label assignment unit 102, an object detection annotation unit 103, an object detection learning unit 104, a work determination parameter calculation unit 105a, a joint position estimation unit 108, a joint position work learning unit 109, and a work determination unit 107a. In addition, the work determination unit 107a has an object detection unit 1071, a moving body detection unit 1072, and a joint position work estimation unit 1073. In addition, the storage unit 20 has an image data storage unit 201, a work registration storage unit 202, and an input data storage unit 203.

[0122] The storage unit 20 , the image data storage unit 201 , the job registration storage unit 202 , and the input data storage unit 203 have the same functions as those of the first embodiment.

[0123] The job registration unit 101 , the job label assignment unit 102 , the object detection annotation unit 103 , and the object detection learning unit 104 have the same functions as those of the first embodiment.

[0124] The object detection unit 1071 and the moving object detection unit 1072 have functions equivalent to those of the object detection unit 1071 and the moving object detection unit 1072 in the first embodiment.

[0125] The joint position estimation unit 108 estimates the joint position information related to the joint position of the operator for each frame image (still image) of the image data to which the work label is assigned and stored in the input data storage unit 203. In addition, the frame image may be extracted from the image data at appropriate intervals. For example, when the frame rate of the image data is 60 fps, the frame image may be extracted at about 24 fps.

[0126] Specifically, the joint position estimation unit 108 uses a well-known method (for example, Kosuke Sugano, Kenta Oku, Kyoji Kawagoe, "Action Detection and Classification Techniques Based on Multidimensional Time Series Data", DEIM Forum 2016G4-5, or Shohei Uezono and Toshiji Ono, "Feature Extraction of Multimodal Sequence Data Using LSTM Autoencoder", Artificial Intelligence Society Research Group Materials, SIG-KBS-B802-01, 2018) to estimate the time series data such as the coordinates and angles of the joints of the operator's hand or wrist as joint position information for each frame image (still image) of the image data assigned a work label and stored in the input data storage unit 203.

[0127] Fig.12 is a diagram showing an example of joint position information in a frame image. Fig.12 , the joint position information when the operator is performing friction is shown.

[0128] The joint position operation learning unit 109 performs machine learning using, for example, the joint position information estimated by the joint position estimating unit 108 as input data and the operation label assigned by the operation label assigning unit 102 as label data to generate a joint position operation estimation model for estimating the operator's operation.

[0129] For example, the joint position task learning unit 109 generates a joint position task estimation model so that Fig.12 The joint position information of the operator's right hand is as follows Fig.13 When there is a reciprocating motion once every 0.3 seconds or so as shown, it is determined that friction is occurring.

[0130] Furthermore, the joint position work learning unit 109 may generate a rule base based on the joint position information estimated by the joint position estimating unit 108 and the work label assigned by the work label assigning unit 102 .

[0131] The job judgment parameter calculation unit 105a calculates the judgment criterion (parameters) so as to minimize the error with the job label based on the value related to the accuracy of object detection in job judgment using the object detection model and the classification probability of the job estimated based on the joint position in job judgment using the joint position job estimation model.

[0132] Specifically, the work determination parameter calculation unit 105a, for example, is similar to the work determination parameter calculation unit 105 of the first embodiment. Figure 2 The operation determination parameter calculation unit 105a inputs the annotated frame image data of other image data to which the operation label is assigned stored in the input data storage unit 203 to the object detection model, detects the tool (object), and obtains the value related to the object detection. The operation determination parameter calculation unit 105a calculates the initial value of the parameter used as the judgment standard for each operation registered in the operation table. Figure 2 In addition, the work judgment parameter calculation unit 105a estimates the joint position information of the operator for each frame image (still image) of the same other image data, inputs the estimated joint position information into the joint position work estimation model, thereby estimating the operator's work and obtaining the classification probability estimated based on the joint position.

[0133] Then, the job judgment parameter calculation unit 105a sets the weight coefficient of the classification probability of the job inferred based on the joint position to a, and sets the weight coefficient of the value related to the accuracy of object detection to b, so as to minimize the error between the classification probability of the job calculated using the following formula (1) and the correct job label, and calculates the value of the parameter (judgment criterion) through Bayesian optimization, etc.

[0134] Classification probability of work = a (classification probability of work estimated from joint positions) + b (value related to object detection accuracy) (1)

[0135] Here, the parameters include, for example, the number of seconds X for performing the work X seconds after the object is detected, the weight a of the classification probability of the work estimated from the joint positions, and the weight b of the value related to the accuracy of object detection.

[0136] The work determination parameter calculation unit 105 a outputs the calculated parameter to a work determination unit 107 a described later and sets the parameter.

[0137] The work determination unit 107 a determines the work of the worker in the video data newly input from the camera 2 using the object detection model, the joint position work estimation model, and the set parameters (determination criteria).

[0138] Specifically, the work determination unit 107a inputs, for example, a frame image (still image) of video data newly input from the camera 2 to the object detection model in the object detection unit 1071 and the moving body detection unit 1072. The work determination unit 107a detects the tool (object), Figure 2 The operation determination unit 107a determines the operation of the operator based on the operation table and parameters, and obtains a value related to the accuracy of object detection. In addition, the operation determination unit 107a estimates the joint position information of the operator for each frame image (still image) of the same newly input image data, and inputs the estimated joint position information into the joint position operation estimation model in the joint position operation estimation unit 1073 described later. The operation determination unit 107a obtains the estimation result of the operator's operation and the classification probability of the operation estimated based on the joint position from the joint position operation estimation unit 1073 described later.

[0139] Then, the work judgment unit 107a calculates the classification probability of the work based on the obtained classification probability of the work inferred from the joint position and the value related to the accuracy of object detection, the set parameters, and formula (1), and judges the operator's work based on the calculated classification probability and the detection result of the moving body detection unit 1072.

[0140] The joint position work estimation unit 1073 has a joint position work estimation model generated by the joint position work learning unit 109, inputs the joint position information estimated by the work judgment unit 107a into the joint position work estimation model, and outputs the estimation result of the operator's work and the classification probability of the work estimated based on the joint position to the work judgment unit 107a.

[0141] <Parameter Calculation Processing of Work Analysis Device 1A>

[0142] Next, the operation of the parameter calculation process of the work analysis device 1A according to the second embodiment will be described.

[0143] Fig.14 1A is a flowchart for explaining the parameter calculation process of the work analysis device 1A. Fig. 9 The processing from step S1 to step S3 is the same, and the detailed description is omitted.

[0144] In step S34 , the joint position estimating unit 108 estimates the joint position information of the worker for each frame image (still image) of the video data to which the work label is added and stored in the input data storage unit 203 .

[0145] In step S35 , the joint position work learning unit 109 performs machine learning using the joint position information estimated in step S34 as input data and the work label assigned in step S31 as label data, and generates a joint position work estimation model for estimating the worker's work.

[0146] In step S36, the work determination parameter calculation unit 105a inputs the annotated frame image data of other image data to which the work label is assigned stored in the input data storage unit 203 into the object detection model to obtain the detected tool (object) and the value related to the accuracy of object detection.

[0147] In step S37 , the work determination parameter calculation unit 105 a determines the work of the worker based on the object detection result in step S36 and the work table.

[0148] In step S38, the work determination parameter calculation unit 105a estimates the joint position information of the worker based on the frame image (still image) of the same other video data.

[0149] In step S39 , the work determination parameter calculation unit 105 a inputs the joint position information estimated in step S38 to the joint position work estimation model, and acquires the estimation result of the worker's work and the classification probability estimated from the joint position.

[0150] In step S40, the work judgment parameter calculation unit 105a calculates the value of the parameter (judgment criterion) by Bayesian optimization or the like so as to minimize the error between the classification probability of the work calculated by equation (1) and the correct work label.

[0151] <Analysis Processing by Work Analysis Device 1A>

[0152] Next, the operation of the analysis process of the work analysis device 1A according to the second embodiment will be described.

[0153] Fig.15 This is a flowchart for explaining the analysis process of the work analysis device 1A. The flow shown here is repeatedly executed while the video data is input from the camera 2.

[0154] In step S51 , the object detection unit 1071 inputs a frame image (still image) of video data newly input from the camera 2 to the object detection model, detects a tool (object), and obtains a value related to the accuracy of object detection.

[0155] In step S52 , the moving object detection unit 1072 detects a moving object such as a worker or a tool based on changes such as changes in brightness of pixels in a designated image region of each frame image (still image) of video data newly input from the camera 2 .

[0156] In step S53 , the joint position work estimating unit 1073 estimates the worker's joint position information for each frame image (still image) of the newly input video data.

[0157] In step S54 , the joint position work estimating unit 1073 inputs the joint position information estimated in step S53 into the joint position work estimating model to estimate the work of the worker and obtains the classification probability of the work estimated from the joint positions.

[0158] In step S55, the work determination unit 107a calculates the classification probability of the work based on the classification probability of the work inferred from the joint positions obtained in steps S51 and S54 and the value related to the accuracy of object detection, the detection result of the moving body in step S52, the set parameters, and formula (1), and determines the operator's work based on the calculated classification probability.

[0159] As described above, the work analysis device 1A according to the second embodiment can automatically adjust the determination criteria in order to accurately determine the work. In other words, the user only needs to label the work and annotate the object to automatically calculate the optimal parameters.

[0160] The second embodiment has been described above.

[0161] As mentioned above, although the 1st embodiment and the 2nd embodiment were demonstrated, the work analysis apparatus 1, 1A is not limited to the said embodiment, and includes deformation|transformation, improvement, etc. within the range which can achieve an objective.

[0162] <Modification 1>

[0163] In the first and second embodiments, the work analysis devices 1 and 1A are connected to one camera 2 , but the present invention is not limited thereto. For example, the work analysis devices 1 and 1A may be connected to two or more cameras 2 .

[0164] <Modification 2>

[0165] In addition, for example, in the above-mentioned embodiment, the work analysis device 1, 1A has all the functions, but it is not limited to this. For example, the server may also include part or all of the work registration unit 101, the work label assignment unit 102, the object detection annotation unit 103, the object detection learning unit 104, the work determination parameter calculation unit 105, the object detection annotation suggestion unit 106, the work determination unit 107, the object detection unit 1071, and the moving body detection unit 1072 of the work analysis device 1, or include part or all of the work registration unit 101, the work label assignment unit 102, the object detection annotation unit 103, the object detection learning unit 104, the work determination parameter calculation unit 105a, the joint position estimation unit 108, the joint position work learning unit 109, the work determination unit 107a, the object detection unit 1071, the moving body detection unit 1072, and the joint position work estimation unit 7 of the work analysis device 1A. In addition, the functions of the work analysis devices 1, 1A may also be realized by using a virtual server function or the like on the cloud.

[0166] Furthermore, the work analysis apparatus 1 , 1A may be a distributed processing system in which the functions of the work analysis apparatus 1 , 1A are appropriately distributed to a plurality of servers.

[0167] <Variation 3>

[0168] Furthermore, for example, in the above-described embodiment, the work analysis device 1A does not include the object detection comment suggesting unit 106 , but may include the object detection comment suggesting unit 106 .

[0169] Thus, when the accuracy of work determination is insufficient, the work analysis device 1A can automatically suggest frames in a moving image that can improve the accuracy of work determination if annotation is performed.

[0170] In addition, each function included in the work analysis device 1 and 1A in the first embodiment and the second embodiment can be realized by hardware, software, or a combination thereof. Here, realization by software means realization by a computer reading and executing a program.

[0171] The program can be stored using various types of non-transitory computer readable media (Non-transitorycomputer readable medium) and provided to the computer. Non-transitory computer readable media include various types of tangible recording media (Tangible storage medium). Examples of non-transitory computer readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives), optical magnetic recording media (e.g., optical magnetic disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). In addition, the program can also be provided to the computer via various types of transient computer readable media (Transitory computerreadable medium). Examples of transient computer readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer readable media can provide the program to the computer via a wired communication path such as an electric wire and an optical fiber or a wireless communication path.

[0172] Furthermore, the steps for describing the program stored in the recording medium naturally include processing to be performed in time series in the order thereof, and also include processing that is not necessarily performed in time series but is executed in parallel or individually.

[0173] In other words, the work analysis device of the present disclosure can take various embodiments having the following structures.

[0174] (1) The job analysis device 1 disclosed in the present invention is a job analysis device for analyzing the job of an operator, and comprises: a job label assignment unit 102, which assigns a job label representing the operator's job to image data containing the operator's job; an object detection annotation unit 103, which annotates the image data assigned the job label with an object associated with the operator's job; an object detection learning unit 104, which generates an object detection model for object detection based on the image data of the object annotated by the object detection annotation unit 103; an object detection unit 1071, which detects the object from the image data using the object detection model; a job judgment parameter calculation unit 105, which performs job judgment on the image data assigned the job label and calculates a judgment criterion that minimizes the error with the assigned job label; and a job judgment unit 107, which uses the object detection model and the judgment criterion to judge the operator's job in newly input image data.

[0175] According to the work analysis device 1, the determination criterion can be automatically adjusted in order to determine the work with high accuracy.

[0176] (2) According to the job analysis device 1 described in (1), there may be an object detection annotation suggestion unit 106, which uses the judgment criteria calculated by the job judgment parameter calculation unit 105 to perform job judgment on the image data assigned with the job label, and suggests frame images for annotation based on the judgment result of the job judgment.

[0177] Thus, when the accuracy of work determination is insufficient, the work analysis device 1 can automatically suggest frame images in a moving image that can improve the accuracy of work determination if annotation is performed.

[0178] (3) The work analysis device 1A according to (1) or (2) may include: a joint position estimating unit 108, which estimates joint position information related to the joint position of the operator; a joint position work learning unit 109, which generates a joint position work estimation model for estimating the work of the operator based on the joint position information estimated by the joint position estimating unit 108 and the work label information assigned by the work label assigning unit 102; and a joint position work estimation unit 1073, which estimates the work according to the joint position information based on the joint position work estimation model generated by the joint position work learning unit 109, a work judgment parameter calculation unit 105a, which calculates the judgment criterion based on the value related to the accuracy of object detection in the work judgment using the object detection model and the classification probability of the work estimated according to the joint position in the work judgment using the joint position work estimation model, so as to minimize the error with the work label, and a work judgment unit 107a uses the object detection model, the joint position work estimation model and the judgment criterion to judge the work of the operator in the newly input image data.

[0179] Thus, the work analysis device 1A can achieve the same effect as (1).

[0180] (4) The work analysis device 1, 1A described in any one of (1) to (3) may further include a moving body detection unit 1072, which detects a moving body in newly input image data, and the work determination unit 107, 107a determines whether the operator's work is continuing based on the time interval between the moving body detection unit 1072 detecting the moving body.

[0181] Thus, the work analysis devices 1 and 1A can determine the work of the worker with higher accuracy.

[0182] (5) According to the work analysis device 1, 1A described in any one of (1) to (4), the judgment criterion can include at least a threshold value that can be estimated as the duration of the work using the tool (object) after the tool (object) is detected, and a value related to the accuracy of object detection.

[0183] Thus, the work analyzing devices 1 and 1A can accurately determine the work of the operator even when the tool (object) is not detected.

[0184] Description of Reference Numerals

[0185] 1. 1A operation analysis device

[0186] 2 Cameras

[0187] 10. 10a Control unit

[0188] 101 Operation Registration Department

[0189] 102 Job Labeling Department

[0190] 103 Object Detection Annotation Department

[0191] 104 Object Detection Learning Unit

[0192] 105, 105a: Operation determination parameter calculation unit

[0193] 106 Object Detection Annotation Suggestion Department

[0194] 107, 107a operation determination unit

[0195] 1071 Object Detection Unit

[0196] 1072 Moving Body Detection Unit

[0197] 1073 Joint Position Estimation Unit

[0198] 108 Joint position estimation unit

[0199] 109 Joint Position Operation Learning Department

[0200] 20 Storage unit

[0201] 201 Image Data Storage Department

[0202] 202 Job registration storage unit

[0203] 203 is an input data storage unit.

Claims

1. A work analysis device for analyzing a worker's work, It is characterized in that The work analysis device comprises: a work label assigning unit that assigns a work label indicating the work of the operator to the image data including the work of the operator; an object detection and annotation unit that annotates the image data to which the work label is assigned with an object associated with the work of the worker; an object detection learning unit that generates an object detection model for performing object detection based on the image data of the object annotated by the object detection annotation unit; an object detection unit, which detects the object from the image data using the object detection model; a job determination parameter calculation unit that performs job determination on the image data to which the job label is assigned, and calculates a determination criterion that minimizes an error with the assigned job label; as well as The work determination unit determines the work of the worker in the newly input image data using the object detection model and the determination criterion.

2. The work analysis device according to claim 1, It is characterized in that The job analysis device includes an object detection annotation suggestion unit, which uses the judgment criterion calculated by the job judgment parameter calculation unit to perform job judgment on the image data assigned the job label, and suggests frame images for annotation based on the judgment result of the job judgment.

3. The work analysis device according to claim 1 or 2, It is characterized in that The work analysis device comprises: a joint position estimating unit that estimates joint position information related to the joint position of the operator; a joint position work learning unit that generates a joint position work estimation model for estimating the work of the worker based on the joint position information estimated by the joint position estimating unit and the work label information assigned by the work label assigning unit; and a joint position task estimating unit that estimates a task from the joint position information based on the joint position task estimating model generated by the joint position task learning unit, The work judgment parameter calculation unit calculates the judgment criterion in a manner that minimizes the error with the work label based on a value related to the accuracy of object detection in the work judgment using the object detection model and a classification probability of the work estimated based on the joint position in the work judgment using the joint position work estimation model, The work determination unit determines the work of the worker in the newly input image data using the object detection model, the joint position work estimation model, and the determination criterion.

4. The work analysis device according to any one of claims 1 to 3, It is characterized in that The work analysis device further includes a moving body detection unit that detects a moving body in the newly input image data. The work determination unit determines whether the work of the worker is continuing based on an interval of time during which the moving object detection unit detects the moving object.

5. The work analysis device according to any one of claims 1 to 4, It is characterized in that The determination criterion includes at least a time that can be estimated as a duration of a work using the object after the object is detected, and a threshold value related to the accuracy of object detection.