Accuracy deterioration detection device, method, program, and accuracy deterioration detection system

Abstract

To alleviate deterioration in task estimation accuracy.SOLUTION: An accuracy deterioration detection device according to an embodiment of the present invention comprises: a video acquisition unit for acquiring video footage of a worker; a task estimation unit for estimating each task performed by the worker from the video footage using a trained model; an accuracy deterioration detection unit for detecting tasks with deteriorated estimation accuracy; and a notification unit for notifying a user of the tasks with deteriorated estimation accuracy.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an accuracy degradation detection device, method, program, and accuracy degradation detection system. [Background technology]

[0002] A technique for estimating a worker's work using a trained model based on a video of the worker has been known. For example, in the case of assembling equipment, various tasks are estimated by inputting a video of the worker's hands into a trained model. Summary of the Invention [Problem to be solved by the invention]

[0003] However, if the layout of the workplace is changed from the layout of the training data for the trained model (for example, if the location of tools such as screwdrivers is changed), the physical movements of the worker's hands and other parts of the body will change. In other words, the physical movements of the worker's hands and other parts of the body before the layout change will be different from the physical movements of the worker's hands and other parts of the body after the layout change. Therefore, if video footage of the worker after the layout change is used, the estimation accuracy of the trained model will decrease.

[0004] Therefore, an object of the present invention is to improve the decrease in accuracy of task estimation. [Means for solving the problem]

[0005] An accuracy degradation detection device according to one embodiment of the present invention comprises an image acquisition unit that acquires video footage of a worker, an activity estimation unit that estimates each activity of the worker from the video using a trained model, an accuracy degradation detection unit that detects an activity for which the accuracy of the estimation has decreased, and a notification unit that notifies the user of an activity for which the accuracy of the estimation has decreased. [Effects of the Invention]

[0006] According to the present invention, it is possible to improve the decrease in accuracy of task estimation. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram for explaining the layout (before change) of a work site according to one embodiment of the present invention. [Figure 2] FIG. 10 is a diagram for explaining the layout (after change) of a work site according to one embodiment of the present invention. [Figure 3] FIG. 10 is a diagram for explaining key points according to an embodiment of the present invention. [Figure 4] 1 is a diagram illustrating an overall configuration according to an embodiment of the present invention. [Figure 5] 1 is a hardware configuration diagram of an accuracy degradation detection device according to an embodiment of the present invention. [Figure 6] 1 is a functional block diagram of a system for detecting a decrease in accuracy according to an embodiment of the present invention; [Figure 7] 1 is a flowchart illustrating an overall process according to an embodiment of the present invention. [Figure 8] 10 is a flowchart illustrating a process for detecting a decrease in estimation accuracy according to an embodiment of the present invention. [Figure 9] 10 is a flowchart showing a learning data replacement process according to an embodiment of the present invention. [Figure 10] 10 is an example of a screen displayed on an administrator terminal according to an embodiment of the present invention. [Figure 11] 10 is an example of a screen displayed on an administrator terminal according to an embodiment of the present invention. [Figure 12] 10 is an example of a screen displayed on an administrator terminal according to an embodiment of the present invention. [Figure 13] 10 is an example of a screen displayed on an administrator terminal according to an embodiment of the present invention. [Figure 14] 10 is an example of a screen displayed on an administrator terminal according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] <Summary> The present invention will be outlined below with reference to Figures 1 to 3. For example, the present invention is used in a system that determines whether a worker has assembled a device in the correct procedure.

[0010] In the present invention, each task of a worker is estimated from a video of the worker using a trained model. Specifically, annotations (correct labels) indicating which task is being performed are added to the video of the worker, which is the training data. It is desirable that the layout at the time of estimation is the same as the layout at the time of training (i.e., the layout of the video to which annotations are added).

[0011] Figure 1 shows the layout before the change (i.e., the layout of the training data for the trained model), and Figure 2 shows the layout after the change (i.e., the layout that reduces the estimation accuracy of the trained model). In Figures 1 and 2, during equipment assembly work, the unit that is the target of the work is placed in front of the worker, and the worker is assembling parts using a screwdriver or the like.

[0012] 1 is a diagram for explaining the layout (before modification) of a work site according to an embodiment of the present invention. In FIG. 1, the screwdriver is positioned farther from the worker's hand than in FIG. 2.

[0013] 2 is a diagram for explaining the layout (after modification) of a work site according to one embodiment of the present invention, in which the screwdriver is positioned closer to the worker's hand than in FIG.

[0014] In Figures 1 and 2, the displacement of key points on the worker's body, such as the worker's hands, and the speed at which the key points move are different, which reduces the accuracy of the task estimation.

[0015] 3 is a diagram illustrating key points according to an embodiment of the present invention. Key points (feature points) on the worker's body (e.g., hands) are detected from a video of the worker. The work being performed by the worker is estimated based on the displacement and movement speed of these key points.

[0016] <Overall structure> 4 is a diagram showing the overall configuration of an embodiment of the present invention. The accuracy degradation detection system 1 can include an accuracy degradation detection device 10, an administrator terminal 20, and a camera 30. The accuracy degradation detection device 10 is communicably connected to the administrator terminal 20 and the camera 30 via an arbitrary network.

[0017] <<Accuracy degradation detection device>> The accuracy degradation detection device (e.g., a server) 10 estimates each task of a worker from a video of the worker using a trained model, and when it detects a task for which the estimation accuracy has decreased, it notifies the manager terminal 20 of the task for which the estimation accuracy has decreased. The accuracy degradation detection device 10 is one or more computers.

[0018] <<Administrator terminal>> The manager terminal 20 is a terminal (for example, a personal computer, a smartphone, etc.) operated by a manager (for example, a manager of equipment such as the camera 30). When the manager terminal 20 receives a notification of an operation for which the estimation accuracy has decreased from the accuracy decrease detection device 10, the manager replaces the learning data only for the operation for which the estimation accuracy has decreased and re-learns it, or changes the layout of the work site where the operation for which the estimation accuracy has decreased is being performed.

[0019] <<Camera>> The camera 30 captures an image of a worker at a work site where work is being performed, and transmits the captured image of the worker to the accuracy degradation detection device 10. For example, the camera 30 is a fixed camera such as a fixed-point camera.

[0020] The devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the accuracy degradation detection device (e.g., server) 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0021] <Hardware configuration> 5 is a hardware configuration diagram of an accuracy degradation detection device (for example, a server) 10 according to one embodiment of the present invention. The same applies to the administrator terminal 20.

[0022] As shown in Figure 5, the accuracy degradation detection device 10 is constructed by a computer, and as shown in Figure 5, it is equipped with a CPU 1001, a ROM 1002, a RAM 1003, a HD 1004, an HDD (Hard Disk Drive) controller 1005, a display 1006, an external device connection I / F (Interface) 1007, a network I / F 1008, a data bus 1009, a keyboard 1010, a pointing device 1011, a DVD-RW (Digital Versatile Disk Rewritable) drive 1013, and a media I / F 1015.

[0023] Of these, the CPU 1001 controls the overall operation of the accuracy degradation detection device 10. The ROM 1002 stores programs used to drive the CPU 1001, such as an IPL. The RAM 1003 is used as a work area for the CPU 1001. The HD 1004 stores various data, such as programs. The HDD controller 1005 controls the reading and writing of various data from and to the HD 1004 under the control of the CPU 1001. The display 1006 displays various information, such as a cursor, menu, window, text, or image. The external device connection I / F 1007 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 1008 is an interface for data communication using a communication network. The bus line 1009 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 1001, shown in FIG. 5.

[0024] The keyboard 1010 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 1011 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 1013 controls reading and writing of various data from a DVD-RW 1012, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 1015 controls reading and writing (storing) of data from a recording medium 1014, such as a flash memory.

[0025] <Functional configuration> FIG. 6 is a functional block diagram of an accuracy degradation detection system 1 according to one embodiment of the present invention.

[0026] The accuracy degradation detection device 10 can include a video acquisition unit 101, a keypoint detection unit 102, a task estimation unit 103, an accuracy degradation detection unit 104, a notification unit 105, an accuracy degradation detection interval setting unit 106, a video storage unit 107, an accuracy storage unit 108, a training data replacement processing unit 109, a replacement target task storage unit 110, an annotation storage unit 111, and a training data storage unit 112. By executing a program, the accuracy degradation detection device 10 can function as the video acquisition unit 101, the keypoint detection unit 102, the task estimation unit 103, the accuracy degradation detection unit 104, the notification unit 105, the accuracy degradation detection interval setting unit 106, and the training data replacement processing unit 109.

[0027] The administrator terminal 20 can include a display unit 201. The administrator terminal 20 can function as the display unit 201 by executing a program.

[0028] <<Accuracy degradation detection device>> The video acquisition unit 101 acquires video of a worker at a work site where work is being performed from the camera 30. The video acquisition unit 101 stores the video of the worker in the video storage unit 107.

[0029] The key point detection unit 102 detects key points (feature points) of the worker's body (for example, hands) from the video acquired by the video acquisition unit 101. Specifically, when a video of the worker is input, the key point detection unit 102 estimates the key points of the worker's body using a trained model that outputs key points of the worker's body.

[0030] The task estimation unit 103 estimates each task of the worker from the video using the trained model. Specifically, the task estimation unit 103 estimates the task being performed by the worker (for example, task A is being performed, task B is being performed, task C is being performed, etc.) based on the displacement and movement speed of the keypoints detected by the keypoint detection unit 102.

[0031] Specifically, when the key points of the worker's body are input, the task estimation unit 103 estimates the task being performed by the worker using a trained model that outputs the task being performed by the worker. The task estimation unit 103 stores the accuracy of the task estimation (i.e., the accuracy of the trained model) in the accuracy storage unit 108.

[0032] [Estimation accuracy] Here, the accuracy of estimation will be described. The accuracy of estimation may be any index that indicates the accuracy of a trained model that estimates the work being performed by a worker from key points on the worker's body. For example, the accuracy of estimation is the confidence level of the trained model.

[0033] The accuracy degradation detection unit 104 detects tasks for which the estimation accuracy has decreased, by referring to the accuracy storage unit 108. For example, the accuracy degradation detection unit 104 calculates the accuracy (e.g., average value) for the most recent predetermined period (an interval set by the accuracy degradation detection interval setting unit 106 (e.g., the most recent hour)) for each task and the accuracy (e.g., average value) for a previous predetermined period (an interval set by the accuracy degradation detection interval setting unit 106 (e.g., the previous hour)), and determines that "tasks for which the estimation accuracy has decreased by more than a threshold value" are tasks for which the estimation accuracy has decreased.

[0034] The notification unit 105 notifies the manager terminal 20 of the work in which the estimation accuracy has decreased, which is detected by the accuracy decrease detection unit 104. For example, the notification unit 105 notifies the manager terminal 20 to retrain the trained model, or to return the layout of the work site where the work in which the estimation accuracy has decreased is being performed to its pre-change state.

[0035] The accuracy degradation detection interval setting unit 106 sets an interval for detecting a degradation in accuracy. Specifically, the accuracy degradation detection interval setting unit 106 acquires the interval input to the administrator terminal 20 by the administrator.

[0036] The video storage unit 107 stores video of the worker.

[0037] The accuracy storage unit 108 stores the accuracy (for example, score) of the estimation of each task.

[0038] The learning data replacement processing unit 109 replaces the learning data only for the task for which the estimation accuracy has decreased. For example, the learning data replacement processing unit 109 retrains the trained model (specifically, a trained model that outputs the task being performed by the worker when key points on the worker's body are input) using a video of the task after the estimation accuracy of the task has decreased and to which annotations (annotations acquired from the administrator terminal 20) have been added.

[0039] The replacement target task storage unit 110 stores information about tasks (for example, task A, task B, task C, etc.) selected by the administrator and targeted for replacement of learning data.

[0040] The annotation storage unit 111 stores information about annotations that an administrator has assigned to a video (specifically, annotations (correct labels) that indicate which tasks are being performed).

[0041] The learning data storage unit 112 stores learning data used for relearning a learned model.

[0042] <<Administrator terminal>> The display unit 201 acquires and displays various data from the accuracy degradation detection device 10.

[0043] <Method> 7 is a flowchart showing the overall process according to one embodiment of the present invention. Assume that the camera 30 is capturing an image of a worker at a work site where work is being performed.

[0044] In step 101 (S101), the image acquisition unit 101 acquires, from the camera 30, an image of a worker.

[0045] In step 102 (S102), the key point detection unit 102 detects key points (feature points) of the worker's body (for example, hands) from the video captured in S101.

[0046] In step 103 (S103), the task estimation unit 103 estimates the task being performed by the worker based on the displacement and movement speed of the key points detected in S102 (for example, task A being performed, task B being performed, task C being performed, etc.).

[0047] In step 104 (S104), the accuracy degradation detection unit 104 detects an operation in which the estimation accuracy has decreased. The process of detecting the estimation accuracy degradation will be described in detail later with reference to FIG.

[0048] In step 105 (S105), the notification unit 105 notifies the manager terminal 20 of the work for which the estimation accuracy has decreased, which was detected in S104 (for example, the screen in FIG. 11). When the manager terminal 20 receives the notification of the work for which the estimation accuracy has decreased from the accuracy decrease detection device 10, the manager replaces the learning data only for the work for which the estimation accuracy has decreased and re-learns it (step 106-1), or changes the layout of the work site where the work for which the estimation accuracy has decreased is being performed (step 106-2).

[0049] In step 106-1 (S106-1), the learning data replacement processing unit 109 replaces learning data only for the task for which the estimation accuracy has decreased, and performs re-learning. The learning data replacement process will be described in detail later with reference to FIG.

[0050] In step 106-2 (S106-2), the layout of the work site where the work with the reduced estimation accuracy is being performed is changed (i.e., the layout is returned to the state before the layout change). The manager can infer what kind of layout change has been made from the work with the reduced estimation accuracy.

[0051] FIG. 8 is a flowchart showing a process for detecting a decrease in estimation accuracy according to an embodiment of the present invention.

[0052] In step 201 (S201), the accuracy degradation detection unit 104 determines whether or not the interval (e.g., x hours) set by the accuracy degradation detection interval setting unit 106 has elapsed. If the interval has elapsed, the process proceeds to step 202, and if the interval has not elapsed, the process waits until the interval set by the accuracy degradation detection interval setting unit 106 has elapsed.

[0053] In step 202 (S202), the accuracy reduction detection unit 104 acquires the estimation accuracy of each task (for example, the estimation accuracy of task A, the estimation accuracy of task B, the estimation accuracy of task C, etc.) from the accuracy storage unit .

[0054] In step 203 (S203), the accuracy degradation detection unit 104 determines whether or not there is an operation in which the estimation accuracy has decreased (for example, an operation in which the estimation accuracy has decreased more than a threshold value). If there is an operation in which the estimation accuracy has decreased, the process proceeds to step 204, and if there is no operation in which the estimation accuracy has decreased, the process returns to step 201.

[0055] In step 204 (S204), the notification unit 105 notifies the manager terminal 20 of the work in which the estimation accuracy has decreased, which was detected in S204.

[0056] FIG. 9 is a flowchart showing the learning data replacement process according to one embodiment of the present invention.

[0057] In step 301 (S301), the learning data replacement processing unit 109 causes the manager terminal 20 to display a screen (for example, the screen in FIG. 12) showing the estimation accuracy of each task before and after the estimation accuracy decreased (i.e., S104 in FIG. 7, S203 in FIG. 8).

[0058] In step 302 (S302), the learning data replacement processing unit 109 acquires from the administrator terminal 20 information on the task for which the learning data is to be replaced (for example, task A, task B, task C, etc.), which the administrator selected on the screen of S301.

[0059] In step 303 (S303), the learning data replacement processing unit 109 stores in the replacement target task storage unit 110 information on the tasks for which the learning data is to be replaced, selected in S302 (for example, task A, task B, task C, etc.).

[0060] In step 304 (S304), the learning data replacement processing unit 109 causes the administrator terminal 20 to display a screen (for example, the screen in FIG. 13) showing a list of videos included in the video of the work for which the learning data selected in S302 is to be replaced.

[0061] In step 305 (S305), the learning data replacement processing unit 109 determines whether or not the annotation has been completed. If it has been completed, the process proceeds to step 309; if it has not been completed, the process proceeds to step 306.

[0062] In step 306 (S306), the learning data replacement processing unit 109 acquires from the administrator terminal 20 information on the video (video to which annotations are to be added) for which learning data is to be replaced, which the administrator selected on the screen of S304.

[0063] In step 307 (S307), the learning data replacement processing unit 109, in accordance with instructions from the administrator terminal 20 (for example, the screen in FIG. 14), assigns annotations (annotations (correct labels) indicating which tasks are being performed in each frame; information stored in the replacement task memory unit 110 in S303) to each frame included in the video.

[0064] In step 308 (S308), the learning data replacement processing unit 109 stores the annotation information added in S307 in the annotation storage unit 111.

[0065] In step 309 (S309), the training data replacement processing unit 109 stores the annotated video in the training data storage unit 112. Thereafter, the trained model is retrained using the annotated video (i.e., the video of the task after the accuracy of task estimation has decreased and the annotated video).

[0066] <User Interface> Screens displayed on the administrator terminal 20 will be described with reference to FIGS.

[0067] Fig. 10 is an example of a screen displayed on the administrator terminal 20 according to one embodiment of the present invention. Fig. 10 is a screen for setting the interval for detecting a decrease in accuracy. The administrator can input the interval for detecting a decrease in accuracy (for example, every x hours, every y hours, every z days, etc.) on the screen of Fig. 10. The administrator terminal 20 transmits information about the interval for detecting a decrease in accuracy input by the administrator to the accuracy decrease detection device 10.

[0068] FIG. 11 is an example of a screen displayed on the manager terminal 20 according to one embodiment of the present invention. FIG. 11 is a screen showing the accuracy (e.g., average value) for each task over the most recent predetermined period (the interval set in FIG. 10) and the accuracy (e.g., average value) over the past predetermined period (the interval set in FIG. 10). For example, the display mode of tasks for which the estimation accuracy has decreased (e.g., tasks for which the accuracy has decreased by more than a threshold value) may be different from the display mode of other tasks. The manager can know, on the screen of FIG. 11, tasks for which the estimation accuracy has decreased (i.e., tasks for which the layout of the work site may have been changed).

[0069] Fig. 12 is an example of a screen displayed on the administrator terminal 20 according to one embodiment of the present invention. Fig. 12 is a screen for selecting an operation for which learning data is to be replaced. The administrator can select the operation for which learning data is to be replaced on the screen of Fig. 12. The administrator terminal 20 transmits information about the operation selected by the administrator to the accuracy degradation detection device 10.

[0070] FIG. 13 is an example of a screen displayed on the administrator terminal 20 according to one embodiment of the present invention. FIG. 13 is a screen for selecting a video (a video included in the video of the work selected in FIG. 12; the time when the video was filmed) for which learning data is to be replaced. On the screen of FIG. 13, the administrator can select a video for which learning data is to be replaced. The administrator terminal 20 transmits information about the video selected by the administrator to the accuracy degradation detection device 10. When the complete button is pressed in FIG. 13, the replacement of learning data is completed.

[0071] FIG. 14 is an example of a screen displayed on the administrator terminal 20 according to one embodiment of the present invention. FIG. 14 is a screen for adding annotations to each frame (frames included in the video selected in FIG. 13) for which learning data is to be replaced. On the screen of FIG. 14, the administrator can select the frame for which learning data is to be replaced. The administrator terminal 20 transmits information about the frame selected by the administrator to the accuracy degradation detection device 10. The operation selected in FIG. 12 is added as an annotation. Note that when the selection completion button is pressed on FIG. 14, the screen returns to the screen of FIG. 13.

[0072] <Effects> In this way, in one embodiment of the present invention, the degradation of the accuracy of task estimation can be improved by replacing the learning data only for tasks for which the estimation accuracy has decreased and re-learning. Also, the degradation of the accuracy of task estimation can be improved by changing the layout of the work site where the task for which the estimation accuracy has decreased is being performed.

[0073] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above. [Explanation of symbols]

[0074] 1. Accuracy degradation detection system 10 Accuracy degradation detection device 20 Administrator terminal 30 Camera 101 Video acquisition unit 102 Keypoint detection unit 103 Work Estimation Department 104 Accuracy degradation detection unit 105 Notification Department 106 Accuracy degradation detection interval setting unit 107 Video storage unit 108 Precision memory unit 109 Learning data replacement processing unit 110 Replacement target working memory section 111 Annotation Storage Unit 112 Learning data storage unit 201 Display section [Prior art documents] [Patent documents]

[0075] [Patent Document 1] Japanese Patent Application Publication No. 2019-200241

Claims

1. an image acquisition unit that acquires an image of a worker; an operation estimation unit that estimates each operation of the worker using a trained model from the video; an accuracy reduction detection unit that detects an operation in which the accuracy of the estimation has decreased; a notification unit that notifies the user of the work in which the accuracy of the estimation has decreased; An accuracy degradation detection device comprising:

2. The accuracy degradation detection device according to claim 1 , further comprising a learning data replacement processing unit that replaces learning data only for the task for which the estimation accuracy has decreased.

3. The accuracy degradation detection device according to claim 2 , wherein the training data replacement processing unit retrains the trained model using video of the task to which annotations have been added after the accuracy of the task estimation has decreased.

4. The accuracy degradation detection device according to claim 3 , wherein the learning data replacement processing unit acquires the annotations from an administrator terminal.

5. The accuracy degradation detection device according to claim 1 , wherein the notification unit issues a notification to restore the layout of the work site where the work for which the estimation accuracy has decreased is being performed to its pre-change state.

6. The accuracy degradation detection device according to claim 1 , wherein the notification unit causes a screen indicating the estimation accuracy of each task before and after the estimation accuracy degradation is displayed on an administrator terminal.

7. A method performed by a degradation in accuracy detection device, comprising: A step in which a worker acquires the captured video; A step of estimating each task of the worker from the video using a trained model; Detecting an operation in which the accuracy of the estimation has decreased; notifying the work in which the accuracy of the estimation has decreased; A method comprising:

8. Accuracy degradation detection device, A process of acquiring the video taken by the worker; A process of estimating each task of the worker using a trained model from the video; A process of detecting an operation in which the accuracy of the estimation has decreased; A process of notifying the work in which the accuracy of the estimation has decreased A program to execute.

9. An accuracy degradation detection system including an accuracy degradation detection device and an administrator terminal, The accuracy degradation detection device an image acquisition unit that acquires an image of a worker; an operation estimation unit that estimates each operation of the worker using a trained model from the video; an accuracy reduction detection unit that detects an operation in which the accuracy of the estimation has decreased; a notification unit that notifies the manager terminal of the work in which the accuracy of the estimation has decreased; An accuracy degradation detection system comprising:

Images