Task analysis device
The task analysis device automates the setting of determination criteria for precise task identification by integrating object recognition and joint position estimation, addressing complexity and computational inefficiencies in existing methods.
Patent Information
- Application Number
- DE112022006288
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-12-05
AI Technical Summary
Existing task analysis methods face challenges such as complex and difficult-to-interpret classification models, high computational requirements for object detection, and the need for manual adjustment of task determination criteria, which are time-consuming and lack accuracy verification.
A task analysis device comprising a task label assigning unit, object recognition annotation unit, object detection learning unit, object detection unit, task determination parameter calculation unit, and task determination unit, which automatically sets and derives determination criteria for precise task determination using object recognition models and joint position estimation.
Enables automatic setting of determination criteria for accurate task identification, reducing manual effort and improving detection accuracy by suggesting frames for annotation to enhance model performance.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present invention relates to a task analysis device. BACKGROUND ART
[0002] It's possible to obtain operating data for machine tools or similar devices in a factory, but it hasn't been possible to obtain data related to a worker's task. Accordingly, improving tasks, considering the introduction of a robot, and creating a digital twin or similar for a factory requires visualizing the tasks performed by workers. Therefore, a technique for automatically detecting what a worker has done from a video of a task is important.
[0003] In this regard, there is a known technique that performs machine learning using data to be learned, which includes input data—that is, an image resulting from the capture of a worker's task—and label data for the worker's task indicated by the image. It generates a trained model to identify the task from the image, and uses the trained model to identify which task is being performed in an image to be analyzed. See, for example, Patent Document 1.
[0004] In addition, there is a known technique that identifies the position of a worker's hand from image data having depth acquired by a depth sensor, identifies the position of a target object from image data acquired by a digital camera, and identifies the details of an operation performed by the worker during the task. See, for example, Patent Document 2. Citation listPatent document Patent Document 1: Unopposed Japanese Patent Application, Publication No. 2021-67981 Patent Document 2: PCT International Publication No. WO2017 / 222070 DISCLOSURE OF THE INVENTION Problems to be solved by the invention
[0005] However, a classification model such as the one trained in Patent Document 1 has the problem of being complex and difficult to interpret.
[0006] In addition, to detect a tool (object) used in an image for the purpose of task classification, as in Patent Document 2, a large number of calculations are required to scan the entire image.
[0007] Furthermore, accurately determining a task performed by a worker requires adjusting task determination criteria (parameters) and manually searching through images of various task scenes and annotating them, which takes time and effort. Furthermore, it is unknown whether the accuracy of task determination improves even when manual searching is performed.
[0008] Accordingly, a functionality for automatically setting and deriving a determination criterion (parameter) for the precise determination of a task is desired. Means to solve the problems
[0009] One aspect of a task analysis device according to the present disclosure is a task analysis device for analyzing a task by a worker, comprising: a task label assignment unit configured to assign a task label indicating the worker's task to video data containing the worker's task; an object recognition annotation unit configured to make an annotation for an object related to the worker's task with respect to the video data to which the task label has been assigned; an object recognition learning unit configured to generate an object detection model for performing object detection from video data concerning the object for which the annotation has been made by the object recognition annotation unit;an object detection unit configured to use the object detection model to detect the object from the video data; a task determination parameter calculation unit configured to perform task determination on the video data assigned the task label and calculate a determination criterion for minimizing an error with respect to the assigned task label; and a task determination unit configured to use the object detection model and the determination criterion to determine a task by the worker in newly input video data. Effects of the invention
[0010] Through one aspect it is possible to automatically set and derive a determination criterion (parameter) in order to cause a task to be determined precisely. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a functional block diagram showing an example of a functional configuration of a task analysis system according to a first embodiment; Fig. 2 is a view showing an example of a task table; Fig. 3 is a view showing an example of a user interface for assigning a task label; Fig. 4 is a view showing examples of video data with different states for a rotating tool; Fig. 5 is a view showing an example of determination results in task determination; Fig. 6 is a view showing an example of misdetection; Fig. 7 is a view showing an example of an image region in video data; Fig. 8 is a view showing an example of the operation of a dynamic body detection unit; Fig. 9 is a flowchart for describing parameter calculation by a task analysis device; Fig. 10 is a flowchart for describing the analytical processing by the task analysis device; Fig. 11 is a functional block diagram showing an example of a functional configuration of a task analysis system according to a second embodiment; Fig. 12 is a view showing an example of joint position information in a single image; Fig. 13 is a view showing an example of the operation of a joint position task estimation model; Fig. 14 is a flowchart for describing the parameter calculation processing by a task analysis device; and Fig. 15 is a flowchart for describing the analytical processing by the task analysis device. PREFERRED EMBODIMENT OF THE INVENTION
[0011] Referring to the drawings, a first and a second embodiment of a task analysis device will be described in detail.
[0012] Here, these embodiments share a configuration for assigning a task label indicating a task by a worker in advance to video data (moving image) resulting from the worker's acquisition of a task, annotating the video data to which the task label has been assigned with an object (a tool) related to the task by the worker, and generating an object recognition model that recognizes an object consisting of an annotated object from the video data.
[0013] However, in the first embodiment, when determining a task by a worker, a generated object recognition model is used to perform task determination for a worker's task in video data assigned a task label, and a determination criterion is calculated to minimize an error with respect to the assigned task label. The object recognition model and the calculated determination criterion are used to determine a task by a worker in newly input video data. In contrast, the second embodiment differs from the first embodiment in that: estimating joint position information related to joints of a worker; generating a joint position task estimation model for estimating a task by the worker based on the estimated joint position information and an assigned task label; andCalculating a determination criterion based on a value related to the accuracy of object detection for a task determination using an object detection model and a task classification probability estimated from joint positions in a task determination that uses the model to estimate the joint position for the task, such that an error related to the task label is minimized; and using the object detection model, the model to estimate the joint position for the task, and the determination criterion to determine a task by a worker in newly input video data.
[0014] Hereinafter, the first embodiment will first be described in detail, and then the parts of the second embodiment which are different from the first embodiment will mainly be described. <Erste Ausführungsform>
[0015] Fig. 1 is a functional block diagram showing an example of a functional configuration of a task analysis system according to a first embodiment.
[0016] As in Fig. 1, a task analysis system 100 includes a task analysis device 1 and a camera 2.
[0017] Furthermore, the task analysis device 1 and the camera 2 may be connected to each other via a network (not shown), such as a LAN (Local Area Network) or the Internet. In this case, the task analysis device 1 and the camera 2 may each be equipped with a communication unit (not shown) to communicate with each other via the corresponding connection. Note that the task analysis device 1 and the camera 2 may be directly connected to each other via connection interfaces (not shown) wirelessly or wired.
[0018] In addition, the task analysis device 1 is in Fig. 1 is connected to one camera 2, but can also be connected to two or more cameras 2.
[0019] Camera 2 is a digital camera or similar device that captures a two-dimensional image at a predetermined frame rate (e.g., 30 fps, etc.) resulting from the projection of workers and objects such as tools (not shown) onto a plane perpendicular to the optical axis of camera 2. Camera 2 outputs the captured individual images as video data to task analysis device 1. The video data captured by camera 2 may be a visible light image such as an RGB color image, a grayscale image, or a depth image. <Aufgabe Analysegerät 1>
[0020] The task analysis device 1 is a computer device generally known to those skilled in the art and comprises a control unit 10 and a storage unit 20, as shown in Fig. 1. Furthermore, the control unit 10 includes a task registration unit 101, a task label assignment unit 102, an object recognition annotation unit 103, an object recognition learning unit 104, a task determination parameter calculation unit 105, an object recognition annotation suggestion unit 106, and a task determination unit 107. Furthermore, the task determination unit 107 includes an object recognition unit 1071 and a dynamic body recognition unit 1072.
[0021] The storage unit 20 is a storage medium such as a ROM (Read-Only Memory) or an HDD (Hard Disk Drive). The storage unit 20 stores, among other things, an operating system and an application program executed by the control unit 10, which will be described below. Furthermore, the storage unit 20 includes a video data storage unit 201, a task registration storage unit 202, and an input data storage unit 203.
[0022] The video data storage unit 201 stores video data of workers and objects such as tools captured by the camera 2.
[0023] The task registration storage unit 202 stores, for example, a task table that links a tool (object) detected by the object recognition unit 1071 described later and a corresponding task of a worker, and is registered in advance by the task registration unit 101 described later based on an input operation by a user such as a worker via an input device (not shown) such as a keyboard or a touch panel included in the task analysis device 1.
[0024] Fig. 2 shows an example of a task table.
[0025] As in Fig. As shown in Figure 2, the task table has storage areas for “objects” and “tasks”.
[0026] The “Objects” storage area in the task table stores tool names such as “Rotary tool (e.g. Leutor®)” and “Sandpaper”.
[0027] The “Tasks” storage area in the Tasks table stores tasks such as “Apply Rotate Tool” and “Loop”.
[0028] For example, the input data storage unit 203 stores, among the frames in the video data, a set of frame data resulting from the association of a tool (object) annotated by the object recognition annotation unit 103 (described later) with an image region where the tool appears. This set is to be used as input data when the object recognition learning unit 104 (described later) generates an object recognition model.
[0029] The control unit 10 is something that is generally known to those skilled in the art and has a CPU, a ROM, a RAM (random access memory), a CMOS memory, etc., each of which is configured to communicate with each other via a bus.
[0030] The CPU is a processor that assumes overall control of the task analysis device 1. The CPU reads a system program and an application program stored in the ROM via the bus and controls the entirety of the task analysis device 1 in accordance with a system program and the application program. Consequently, the control unit 10 is configured to implement the functionality for the task registration unit 101, the task label assignment unit 102, the object recognition annotation unit 103, the object recognition learning unit 104, the task determination parameter calculation unit 105, the object recognition annotation suggestion unit 106, and the task determination unit 107, as shown in Fig. 1. Furthermore, the task determination unit 107 is configured to provide functions for the object recognition unit 1071 and the dynamic body recognition unit 1072. Various data such as temporary calculation data or display data are stored in the RAM. Furthermore, the CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory for which a storage state is maintained even when a power supply for the task analysis device 1 is turned off.
[0031] Based on an input operation by a user, e.g., a worker, via the input device (not shown) to the task analysis device 1, the task registration unit 101 assigns the relationship between a tool used (a recognized object) and a task that uses the tool (object) (a task for which there is a desire to recognize) in the Fig. 2 and registers them.
[0032] For example, when a user views video data (moving image data) containing a task of a worker and stored in the video data storage unit 201, the task label assignment unit 102 assigns a task label, a task name for the task performed by the worker, to the video data (moving image data).
[0033] Fig. 3 is a view showing an example of a user interface 30 for assigning a task label.
[0034] As in Fig. 3, the user interface 30 has an area 301 for reproducing video data (moving image) stored in the video data storage unit 201, a playback stop button 302, a slider 303, an area 310 displaying task labels assigned in time series to the video data by the task label assignment unit 102, a rotary tool button 321 displaying a tool for which an annotation is made by the object recognition annotation unit 103 described later, a micro-rotary tool button 322, a sandpaper button 323, a vacuum cleaner button 314, and a completion button 330 for completing the assignment of a task label and / or the annotation of an object.
[0035] Specifically, the task label assignment unit 102 displays, for example, the user interface 30 on a display device (not shown) such as an LCD included in the task analysis device 1, and plays back video data (moving image data) stored by the video data storage unit 201 in the area 301 of the user interface 30. A user operates the playback stop button 302 or the slider 303 via the input device (not shown) for the task analysis device 1, thereby confirming video data. When a worker confirms the task "attach a rotary tool" in the video data during a period from 13:10 to 13:13, the user inputs the task name "attach a rotary tool," and the task label assignment unit 102 assigns the task label "attach a rotary tool" to the video data from 13:10 to 13:13.Furthermore, when a worker confirms a task using a micro-turning tool, the user enters the task name "Apply Micro-Turning Tool" in the video data during a period from 13:13 to 13:18, and the task label assignment unit 102 assigns the task label "Apply Micro-Turning Tool" to the video data from 13:13 to 13:18. Furthermore, when a worker confirms a "grinding" task, the user enters the task name "Grinding" in the video data during a period from 13:18 to 13:20, and the task label assignment unit 102 assigns the task label "Grinding" to the video data from 13:18 to 13:20.Further, in the case of confirmation of a "cleaning" task by a worker in the video data during a period from 13:20 to 13:22, the user inputs the task name "cleaning", and the task label assignment unit 102 assigns the task label "cleaning" to the video data from 13:20 to 13:22.
[0036] The task label assignment unit 102 may display the results of assigning task labels in the region 310 in time series on the display device (not shown) associated with the task analysis device 1. The task label assignment unit 102 outputs the video data to which task labels have been assigned to the object recognition annotation unit 103.
[0037] For example, the object recognition annotation unit 103 annotates a tool (object) associated with a task by a worker with respect to video data to which task labels have been assigned.
[0038] Specifically, the object recognition annotation unit 103 displays, for example, in the area 301 of the user interface 30, frames (still images) in which a tool (object) that is a rotating tool appears from video data from time 13:10 to time 13:13 to which the task label “Attaching a rotating tool” has been assigned, wherein the frames (still images) are separated by a prescribed interval or an interval arbitrarily defined by a user.
[0039] Note that it is desirable to set a prescribed interval or an arbitrarily defined interval such that, for example, about 20 frames (still images) are displayed for each task label.
[0040] This allows the user to work efficiently without having to confirm any number of hours of video data and it is possible to reduce the burden on the user.
[0041] Based on an input from a user, the object recognition annotation unit 103 determines an image area (thick rectangle) for a tool (object) appearing for each frame (still image), as shown in Fig. 3, and notes that the tool (object) is a rotating tool because the rotating tool button 321 or the like was pressed. Note that, for each piece of video data in which the task label "apply micro-rotation tool," "grind," or "clean" has been assigned, with respect to a frame (still image) in which a tool (object) appears, the object recognition annotation unit 103 obtains an image area for the tool (object) that appears and makes an annotation for the tool (object), similarly to the "apply rotating tool" case.
[0042] In a case where an image region where a tool (object) appears and tool (object) annotations for all frames (still images) in video data to which task annotations have been assigned are complete, and the completion button 330 is pressed by a user, the object recognition annotation unit 103 stores in the input data storage unit 203 a set of frame data (hereinafter, may be referred to as “annotated image data”) resulting from associating image regions for frames (still images) obtained from video data (moving image data) for a period in which various tasks were performed (the period from the start of a task to the end of the task) and in which a tool appears (with an assigned timestamp) with tools (objects) for which annotations have been made.
[0043] The object recognition learning unit 104 creates an object recognition model for performing object recognition from video data regarding an object for which an annotation has been made.
[0044] Specifically, the object recognition learning unit 104 generates, for example, an object recognition model that is a trained model such as a neural network using annotated frame image data stored in the input data storage unit 203 as input data, and performs publicly known machine learning using the learning data in which a tool (object) for which an annotation has been made is used as label data. The object recognition learning unit 104 stores the generated object recognition model in the storage unit 20.
[0045] The task determination parameter calculation unit 105 uses the object detection model generated by the object detection learning unit 104 to perform task determination on video data to which a task label has been assigned, and calculates a determination criterion for minimizing an error with respect to the assigned task label.
[0046] Specifically, the task determination parameter calculation unit 105 sets, for example, an initial value for a parameter serving as a determination criterion for each task listed in the task table in Fig. 2. Note that the parameter includes, among other things, a number of seconds X for assuming that a task is performed for X seconds after the detection of an object, a threshold for a value regarding the accuracy of object detection for determining that the task "application of a rotating tool" is being performed, and a threshold for a value regarding the accuracy of object detection for determining that, for example, the task "sanding" is being performed.The number of seconds X for assuming that a task is performed for X seconds after object detection is included in a parameter, whereby the task analysis device 1 can determine that a task is performed using a tool (object) if the tool is detected in the last X seconds - even in a case where it is not possible to detect the tool (object), for example, from video data.
[0047] The task determination parameter calculation unit 105 inputs annotated frame data from separate video data stored in the input data storage unit 203 and assigned a task label to the object detection model, and detects a tool (object). The task determination parameter calculation unit 105 determines the task based on an object detection result and the task table in Fig. 2 and calculates an error between the determined task and the correct task label. Based on the error calculated for all annotated image data, the task determination parameter calculation unit 105 calculates an evaluation index, such as an F1 score, for the parameter value for each task, and uses Bayesian optimization or the like to calculate the parameter value for each task so that the calculated evaluation indices for each task are maximized.
[0048] The object detection annotation suggestion unit 106 uses the parameter (determination criterion) calculated by the task determination parameter calculation unit 105 to perform task determination on video data to which a task label has been assigned, and suggests a frame (still image) to be annotated based on a determination result for the task determination.
[0049] For example, in object detection for a rotating tool, if the training is performed only with video data of a state in which a worker is holding the rotating tool, as in the upper part of Fig. 4, the value for the accuracy of object detection decreases for a rotating tool placed on a workbench or the like, as shown in the lower part of Fig. 4. Accordingly, video data annotated in a variety of different scenes is required, but it would require time and effort for a user to search through such diverse scenes. Therefore, the object detection annotation suggestion unit 106 suggests a single image (still image) to be automatically annotated, as described below.
[0050] Specifically, the object recognition annotation suggestion unit 106 performs task determination by using image data resulting from the association of a tool (object) for which a note has been made with an image area in which the tool appears in separate video data stored in the input data storage unit 203 and to which, for example, a task label has been assigned.
[0051] Fig. Figure 5 is a view showing an example of a determination result for a task determination. The upper part in Fig. Figure 5 shows the correct task labels assigned to each video data in time series. The middle part in Fig. Figure 5 shows determination results for a worker's tasks created by the object recognition annotation suggestion unit 106 with respect to the image data and using the object recognition model and parameters. The lower part in Fig. Figure 5 shows the results of object detection in the image data based on the object detection model.
[0052] As in Fig. As shown in Figure 5, in the time period from 13:40 to 13:43, which has the correct task label “Attaching a rotating tool,” there is a time period in which “Attaching a rotating tool” was not determined (recognized) in a task determination result.
[0053] This is because, for example, there are still frames in which a rotating tool appears, but the rotating tool was not extracted for X seconds to assume that a task is executed for X seconds after the object corresponding to the parameter is detected. To increase the object detection accuracy value, the object detection annotation suggestion unit 106 extracts a still frame in which the rotating tool appears in the separate video data for the period in which "application of the rotating tool" was not determined (detected) in the task determination result.Similar to the object recognition annotation unit 103, the object recognition annotation suggestion unit 106 displays the extracted single image (still image) in the user interface 30, obtains an image region for a rotation tool in the extracted single image (still image) based on an input operation by a user, and makes an annotation for a rotation tool based on pressing the rotation tool button 321. The object recognition annotation suggestion unit 106 stores, in the input data storage unit 203, image data linking an image region for a single image (still image) in which a rotation tool appears (to which a timestamp has been assigned) with the rotation tool for which the annotation was made.
[0054] As in Fig. As shown in Figure 5, the task determination result from 13:43 to 13:50, which has the correct task name "sanding," includes a period in which "applying a rotary tool" was misdetected (misdetected) and a period in which "sanding" was not determined (detected). This misdetermination (misdetection) of "applying a rotary tool" is caused by the misdetection of a rotary tool during object detection with respect to the frame (still image) at time 13:43. Furthermore, "sanding" is not determined (detected) because, regardless of the fact that there are frames (still images) in which sandpaper appears, the sandpaper is not extracted in the number of seconds X, to assume that a task is executed for X seconds after the object is detected in a parameter.
[0055] To increase the object recognition accuracy, the object recognition annotation suggestion unit 106 extracts from the separate video data a frame (still image) in which sandpaper appears near the time 13:43, as well as a frame (still image) in which sandpaper appears during a period for which "sanding" was not determined (detected). The object recognition annotation suggestion unit 106 displays each extracted frame (still image) in the user interface 30, obtains an image area for sandpaper in each frame (still image) based on input operations by a user, and labels the tool (object) as sandpaper because the sandpaper button 323 was pressed.The object recognition annotation suggestion unit 106 stores in the input data storage unit 203 image data linking an image area for a frame (still image) in which sandpaper appears (to which a time stamp has been assigned) with sandpaper for which a note has been made.
[0056] This makes it possible to improve the accuracy of object detection without the user having to spend time and effort searching through different scenes.
[0057] Note that even in a case where the object recognition reliability for a tool (object), that is, a value related to the accuracy of object recognition, is low and equal to or less than a prescribed value (e.g., 20% or the like), the object recognition annotation suggestion unit 106 may extract a frame (still image) in which the tool (object) appears. The object recognition annotation suggestion unit 106 may display an extracted frame (still image) in the user interface 30, obtain an image area for a tool (object) in the extracted frame (still image) based on a user's input operation, and make an annotation for the tool (object).
[0058] Subsequently, the object recognition learning unit 104 performs machine learning using image data containing a single image (still image) labeled with the tool (object) extracted (suggested) by the object recognition annotation suggestion unit 106, and updates the object recognition model. The task determination parameter calculation unit 105 inputs annotated single image data containing a single image (still image) extracted (suggested) by the object recognition annotation suggestion unit 106 into the updated object recognition model to thereby determine a task, and calculates the error between an assigned correct task label and a task determination result. Based on the calculated error, the task determination parameter calculation unit 105 calculates an evaluation index for each task, such asan F1 score for a parameter value, and uses Bayesian optimization or the like to recalculate a parameter value for each task so that the calculated evaluation indices for each task are maximized. For example, the object detection learning unit 104 and the task determination parameter calculation unit 105 repeat the processing until there are zero or less than a prescribed number of frames (still images) extracted (proposed) by the object detection annotation suggestion unit 106. The object detection learning unit 104 outputs the generated object detection model to the object detection unit 1071 described later, and the task determination parameter calculation unit 105 outputs the calculated parameters to the task determination unit 107 described later.
[0059] The task determination unit 107 uses the object recognition model and the set parameters (determination criteria) to determine tasks by a worker in the video data newly fed from the camera 2.
[0060] Specifically, the task determination unit 107 inputs, for example, a single frame (still image) belonging to the video data newly input from the camera 2 into an object recognition model in the object recognition unit 1071 described later and the dynamic body recognition unit 1072 described later. The task determination unit 107 determines a task by a worker based on a tool (object) recognition result output from the object recognition model, a recognition result from the dynamic body recognition unit 1072, the task table in Fig. 2 and the parameter. Note that in a case where it is not possible to recognize a tool (object) using a single frame (still image) from video data, the task determining unit 107 may determine a task by the worker in the single frame based on the parameter regarding the number of seconds X to assume that the task is performed for X seconds after the object detection in a case where the tool (object) is last recognized within X seconds from the single frame.
[0061] In addition, the task determination unit 107 may, for example, determine "no task" for the determination of a task by a worker in a case where a value related to the accuracy of object detection, such as an object detection degree of reliability output by the object detection model belonging to the object detection unit 1071, or a classification probability for a class, is equal to or less than a preset threshold (for example, 70% or the like). For example, in a case where a worker simply touches a workpiece in video data, as in Fig. 6, in a case where it has obtained an object recognition result of “sandpaper” and “reliability level: 40%”, determine that a task of the worker is “not a task” because the reliability level is below a threshold (for example, 70% or the like).
[0062] In this way, misjudgments of tasks can be reduced.
[0063] The object detection unit 1071 has an object detection model generated by the object detection learning unit 104, inputs a frame (still image) of video data newly input from the camera 2 to the object detection model, and outputs an object detection result and a value regarding the accuracy of the object detection, such as a reliability level.
[0064] The dynamic body detection unit 1072 detects a dynamic body such as a worker or a tool based on a change such as a change in the luminance of a pixel in a specific image area from each frame (still image) in video data newly input from the camera 2.
[0065] In particular, the dynamic body recognition unit 1072 may determine that a worker in the video data is performing a task when there is movement, such as a change in the luminance of a pixel in an image area indicated by a thick line rectangle in a single frame (still image), as in Fig. 7 shown.
[0066] In addition, the dynamic body detection unit 1072 can determine that a worker is continuously performing a task if the movement is periodically detected at an interval of X seconds (e.g., 5 seconds or so) or less and is indicated by a dashed rectangle, as shown in Fig. 8. It may be that in a case where a tool (object) such as a rotating tool is detected from a frame (still image) for a time point indicated by shading by the object recognition unit 1071 from a time period in which a movement by a dynamic body is detected, as in the lower part of Fig. 8, the dynamic body recognition unit 1072 determines that a task is performed using the detected tool (object) in the time period.
[0067] In contrast, the dynamic body detection unit 1072 may not determine that a worker is performing a task if no movement is detected for a period of more than X seconds. <Parameterberechnung durch Aufgabenanalysevorrichtung 1>
[0068] Next, the operation of parameter calculation processing by the task analysis device 1 according to the first embodiment will be described.
[0069] Fig. 9 is a flowchart for describing the parameter calculation processing by the task analysis device 1. The flow shown here is executed in a case where a new tool (object) and a new task are registered in the task table by a user such as a worker.
[0070] In step S1, the task label assignment unit 102 displays, on the user interface 30, video data including a worker's task stored in the video data storage unit 201, and assigns a task label indicating the task the worker is performing to the video data based on an input operation by the user.
[0071] In step S2, the object recognition annotation unit 103 obtains, with respect to frames (still images) separated by a prescribed interval or the like for each task label from the video data to which the task label was assigned in step S1, an image region for a tool (object) that appears, and creates an annotation for the tool (object). The object recognition annotation unit 103 stores, in the input data storage unit 203, annotated frame data that associates a tool (object) for which an annotation has been made with an image region in a frame (still image) in which the tool appears (to which a timestamp has been assigned) from video data (moving image data) for the period in which each task was performed (period from the start of the task to the end of the task).
[0072] In step S3, the object recognition learning unit 104 generates an object recognition model for recognizing an object from the annotated image data annotated in step S2.
[0073] In step S4, the task determination parameter calculation unit 105 inputs annotated frame data from separate video data stored in the input data storage unit 203 and assigned a task label to the object recognition model, and recognizes a tool (object).
[0074] In step S5, the task determination parameter calculation unit 105 determines a task by the worker based on the task table and a result of detecting an object in step S4.
[0075] In step S6, the task determination parameter calculation unit 105 calculates, for each task, an error between a correct task label and a result of the determination in step S5.
[0076] In step S7, an evaluation index, e.g., an F1 score for a parameter value, is calculated for each task based on errors calculated with all video data.
[0077] In step S8, the task determination parameter calculation unit 105 uses Bayesian optimization or the like to calculate a parameter for each task so that the evaluation index for each task is maximized.
[0078] In step S9, the object detection annotation suggestion unit 106 uses the parameters (determination criteria) calculated in step S8 to perform task determinations for separate video data to which a task label has been assigned.
[0079] In step S10, the object recognition annotation suggestion unit 106 determines, based on a result of the determination in step S9, whether to suggest a single image (still image) to increase an object recognition accuracy value, such as misdetections or non-detections, for a location where the object recognition accuracy value is low. In a case where a single image (still image) is to be suggested, the processing returns to step S2, and the processing in steps S2 to step S9 is performed again after incorporating the suggested single image (still image).In contrast, in a case where there is no frame image (still image) to propose, the task analysis device 1 sets the object recognition model generated in step S3 to the object recognition unit 1071, sets the parameters calculated in step S8 to the task determination unit 107, and ends the parameter calculation processing. <Analytische Verarbeitung durch Aufgabenanalysevorrichtung 1>
[0080] Next, the operation concerning the analytical processing by the task analysis device 1 according to the first embodiment will be described.
[0081] Fig. 10 is a flowchart for describing the analytical processing by the task analysis device 1. The process shown here is repeatedly executed while video data is input from the camera 2.
[0082] In step S21, the object recognition unit 1071 inputs a frame (still image) derived from video data newly input from the camera 2 into the object recognition model and recognizes a tool (object).
[0083] In step S22, the dynamic body recognition unit 1072 recognizes a dynamic body such as a worker or a tool based on a change such as a change in the luminance of a pixel in a certain image area of each frame (still image) in video data newly input from the camera 2.
[0084] In step S23, the task determining unit 107 determines a task by the worker based on the result of detection of a tool (object) in step S21, the result of detection of a dynamic body in step S22, the set parameters, and the task table.
[0085] As described above, the task analysis device 1 according to the first embodiment can automatically adjust a determination criterion so that a task can be accurately determined. In other words, if a user can label a task and make an annotation for an object, an optimal parameter is automatically calculated.
[0086] In addition, when the accuracy of the task determination is insufficient, the task analysis device 1 can automatically suggest an image in a moving image with which the accuracy of the task determination can be increased when a remark is made.
[0087] The first embodiment was described above. <Zweite Ausführungsform>
[0088] Next, a second embodiment will be described. In the first embodiment, a generated object recognition model is used to perform task determination for a worker's task in video data assigned a task label, and a determination criterion for minimizing an error with respect to the assigned task label is calculated. The object recognition model and the calculated determination criterion are used to determine a worker's task in newly input video data. In contrast, the second embodiment differs from the first embodiment in that: joint position information related to a worker's joints is estimated; a joint position task estimation model is generated for estimating a worker's task based on the estimated joint position information and an assigned task label; andCalculating a determination criterion based on a value related to the accuracy of object detection for a task determination using an object detection model and a task classification probability estimated from joint positions in a task determination that uses the model to estimate the joint position for the task, such that an error related to the task label is minimized; and using the object detection model, the model to estimate the joint position for the task, and the determination criterion to determine a task by a worker in newly input video data.
[0089] As a result, a task analysis device 1A according to the second embodiment can automatically adjust a determination criterion to cause a task to be accurately determined.
[0090] The second embodiment is described below.
[0091] Fig. Fig. 11 is a functional block diagram showing an example of a functional configuration of a task analysis system according to a second embodiment. Note that elements having similar functionality to the elements of the
[0092] Task Analysis System 100 in Fig. 1, the same reference symbols have been added and a detailed description of them is omitted.
[0093] As in Fig. As shown in Figure 11, a task analysis system 100 includes a task analysis device 1A and a camera 2.
[0094] Camera 2 has the same functionality as camera 2 of the first embodiment. <Aufgabenanalysevorrichtung 1A>
[0095] As in Fig. As shown in Figure 11, the task analysis device 1A includes a control unit 10a and a storage unit 20. Furthermore, the control unit 10a includes a task registration unit 101, a task label assignment unit 102, an object recognition annotation unit 103, an object recognition learning unit 104, a task determination parameter calculation unit 105a, a joint position estimation unit 108, a joint position task learning unit 109, and a task determination unit 107a. Furthermore, the task determination unit 107a includes an object recognition unit 1071, a dynamic body recognition unit 1072, and a joint position task estimation unit 1073. Furthermore, the storage unit 20 includes a video data storage unit 201, a task registration storage unit 202, and an input data storage unit 203.
[0096] The storage unit 20, the video data storage unit 201, the task registration storage unit 202, and the input data storage unit 203 have the same functionality as the storage unit 20, the video data storage unit 201, the task registration storage unit 202, and the input data storage unit 203 in the first embodiment.
[0097] Moreover, the task registration unit 101, the task label assignment unit 102, the object recognition annotation unit 103, and the object recognition learning unit 104 have the same functionality as the task registration unit 101, the task label assignment unit 102, the object recognition annotation unit 103, and the object recognition learning unit 104 in the first embodiment.
[0098] Furthermore, the object detection unit 1071 and the dynamic body detection unit 1072 have the same functionality as the object detection unit 1071 and the dynamic body detection unit 1072 in the first embodiment.
[0099] The joint position estimation unit 108 estimates joint position information related to a worker's joint positions for each frame (still image) in the video data stored in the input data storage unit 203 and assigned a task label. Note that the frames can be extracted from the video data at an appropriate interval. For example, if the video data has a frame rate of 60 frames per second, the frames can be extracted at approximately 24 frames per second.
[0100] Specifically, for each frame (still image) in video data stored in the input data storage unit 203 and assigned a task label, the joint position estimation unit 108 uses a publicly known technique (for example, SUGANO, Kousuke, OKU, Kenta, KAWAGOE, Kyoji, “Motion detection and classification method from multidimensional time series data,” DEIM Forum 2016 G4-5, or UEZONO, Shouhei, ONO, Satoshi, “Feature extraction using LSTM Autoencoder for multimodal sequential data,” JSAI Technical Report, SIG-KBS-B802-1, 2018) to estimate, as joint position information, time series data including, among other things, coordinates and an angle for a joint in a worker’s hand, arm, or the like.
[0101] Fig. 12 is a view showing an example of joint position information in a frame image. Fig. 12 shows information about the joint position when a worker is grinding.
[0102] For example, the joint position task learning unit 109 performs machine learning using the joint position information estimated by the joint position estimation unit 108 as input data and the task label assigned by the task label assigning unit 102 as label data, and generates a model for estimating the joint position for a task by a worker.
[0103] For example, if there is a process where the joint position information for the worker’s right hand is in Fig. 12 make a circular movement in 0.3 seconds or similar, as in Fig. 13, the joint position task learning unit 109 generates a model for estimating the joint position task so that it is determined that a grinding operation is being performed.
[0104] Note that, under certain circumstances, the joint position task learning unit 109 may generate a rule base based on the joint position information estimated by the joint position estimation unit 108 and the task label assigned by the task label assignment unit 102.
[0105] Based on a value related to the accuracy of object detection for task determination using an object detection model and a task classification probability estimated from joint positions in task determination using the joint position task estimation model, the task determination parameter calculation unit 105a calculates a determination criterion (parameter) such that an error related to the task label is minimized.
[0106] Specifically, the task determination parameter calculation unit 105a sets, for example, an initial value for each parameter corresponding to a determination criterion for each task listed in the task table in Fig. 2, similar to the task determination parameter calculation unit 105 in the first embodiment. The task determination parameter calculation unit 105a inputs annotated frame data from separate video data stored in the input data storage unit 203 and assigned a task label to the object recognition model, detects a tool (object), and obtains a value related to the object recognition. The task determination parameter calculation unit 105a determines a task by the worker based on the task table in Fig. 2 and a result of object detection. Furthermore, the task determination parameter calculation unit 105a estimates the joint position information of a worker for each frame (still image) in the same separate video data and inputs the estimated joint position information to the joint position task estimation model, thereby estimating a worker's task and obtaining an estimated classification probability from the joint positions.
[0107] The task determination parameter calculation unit 105a sets a weighting coefficient for the task classification probability estimated from the common positions to a, sets a weighting coefficient for a value related to object detection accuracy to b, and uses the following formula (1) to calculate a value for the parameters (determination criteria) using Bayesian optimization or the like so that the error between the calculated task classification probability and a correct task label is minimized. Task classification probability = a (task classification probability estimated from the common positions) + b (value that affects the accuracy of object detection)
[0108] The parameters include, for example, the number of seconds X for the assumption that a task is executed for X seconds from object detection, the weight a for the estimated from the joint positions
[0109] Probability of task classification and the weight b for the value concerning the accuracy of object detection.
[0110] The task determination parameter calculation unit 105a outputs and sets the calculated parameters to the task determination unit 107a described later.
[0111] The task determination unit 107a uses the object detection model, the joint position task estimation model, and the set parameters (determination criteria) to determine tasks by a worker in video data newly input from the camera 2.
[0112] Specifically, the task determination unit 107a inputs, for example, a single frame (still image) belonging to the video data newly input from the camera 2 into an object recognition model in the object recognition unit 1071 and the dynamic body recognition unit 1072. Based on the recognized tool (object), the task table in Fig. 2 and the parameter, the task determination unit 107a determines the worker's task and obtains a value for the object recognition accuracy. Furthermore, the task determination unit 107a estimates the worker's joint position information for each frame (still image) in the same newly input video data, and inputs the estimated joint position information into a joint position task estimation model in the joint position task estimation unit 1073 described later. The task determination unit 107a obtains a worker's task estimation result from the joint position task estimation unit 1073 described later, and a task classification probability estimated from the joint positions.
[0113] The task determination unit 107a calculates the task classification probability from the obtained task classification probability estimated from the common positions, the object recognition accuracy value, the set parameter, and Formula (1), and determines a task by the worker based on the calculated classification probability and a recognition result from the dynamic body recognition unit 1072.
[0114] The joint position task estimation unit 1073 has a joint position task estimation model generated by the joint position task learning unit 109, inputs information about the joint position estimated by the task determination unit 107a to the joint position task estimation model, and outputs to the task determination unit 107a a result of estimation of a task by a worker and a task classification probability estimated from joint positions. <Parameterberechnungsverarbeitung durch Aufgabenanalysegerät 1A>
[0115] Next, the operation of parameter calculation processing by the task analysis device 1A according to the second embodiment will be described.
[0116] Fig. 14 is a flowchart for describing the parameter calculation processing by the task analysis device 1A. Note that the processing in step S31 to step S33 is similar to the processing in step S1 to step S3 in Fig. 9, and a detailed description is omitted.
[0117] In step S34, the joint position estimation unit 108 estimates the information about the joint position of a worker for each frame (still image) in the video data stored in the input data storage unit 203 and to which a task label has been assigned.
[0118] In step S35, the joint position task learning unit 109 performs machine learning using the joint position information estimated in step S34 as input data and the task label assigned in step S31 as label data, and generates a joint position task estimation model for estimating a task by a worker.
[0119] In step S36, the task determination parameter calculation unit 105a inputs annotated frame data from separate video data stored in the input data storage unit 203 and assigned a task label to the object recognition model, recognizes a tool (object), and obtains a value related to the accuracy of object recognition.
[0120] In step S37, the task determination parameter calculation unit 105a determines a task by the worker based on the task table and a result of detecting an object in step S36.
[0121] In step S38, the task determination parameter calculation unit 105a estimates common position information for the worker from frames (still images) from the same separate video data.
[0122] In step S39, the task determination parameter calculation unit 105a inputs the joint position information estimated in step S38 into the joint position task estimation model, and obtains a result of estimation of a task by the worker and a classification probability estimated from the joint positions.
[0123] In step S40, the task determination parameter calculation unit 105a calculates a value for a parameter (determination criterion) using Bayesian optimization or the like so that the error between the task classification probability calculated by formula (1) and the correct task label is minimized. <Analytische Verarbeitung durch Aufgabenanalysevorrichtung 1A>
[0124] Next, the operation of the analytical processing by the task analysis device 1A according to the second embodiment will be described.
[0125] Fig. 15 is a flowchart for describing the analytical processing by the task analysis device 1A. The process shown here is repeatedly executed while video data is input from the camera 2.
[0126] In step S51, the object recognition unit 1071 inputs a frame (still image) derived from video data newly input from the camera 2 into the object recognition model, recognizes a tool (object), and obtains a value related to the accuracy of object recognition.
[0127] In step S52, the dynamic body recognition unit 1072 recognizes a dynamic body such as a worker or a tool based on a change such as a change in the luminance of a pixel in a certain image area of each frame (still image) in video data newly input from the camera 2.
[0128] In step S53, the joint position task estimation unit 1073 estimates joint position information for each frame (still image) in the newly inputted video data for the worker.
[0129] In step S54, the joint position task estimation unit 1073 inputs the joint position information estimated in step S53 into the joint position task estimation model and obtains a worker's estimation result of a task and a task classification probability estimated from the joint positions.
[0130] In step S55, the task determination unit 107a calculates the task classification probability from the task classification probability estimated from the joint positions obtained in step S54, the value related to the object recognition accuracy obtained in step S51, the dynamic body recognition result from step S52, the set parameter, and the formula (1), and determines a task by the worker based on the calculated classification probability.
[0131] As described above, the task analysis device 1A according to the second embodiment can automatically adjust a determination criterion to accurately determine a task. In other words, if a user can label a task and make a comment for an object, an optimal parameter is automatically calculated.
[0132] The second embodiment was described above.
[0133] This concludes the above description of the first and second embodiments, but the task analysis devices 1 and 1A are not limited to the above-described embodiments and include variations, improvements, etc. to an extent that enables the object to be achieved. <Erste Variante>
[0134] In the first and second embodiments, one camera 2 is connected to each of the task analysis devices 1 and 1A, but this is not limiting. For example, two or more cameras 2 may be connected to each of the task analysis devices 1 and 1A. <Zweite Variante>
[0135] As another example, the task analysis devices 1 and 1A have all the functions of the embodiments described above, but this is not a limitation. For example, some or all of the task registration unit 101, the task label assignment unit 102, the object recognition annotation unit 103, the object recognition learning unit 104, the task determination parameter calculation unit 105, the object recognition annotation suggestion unit 106, the task determination unit 107, the object recognition unit 1071, and the dynamic body recognition unit 1072 in the task analysis device 1, or some or all of the task registration unit 101, the task label assignment unit 102, the object recognition annotation unit 103, the object detection learning unit 104, the task determination parameter calculation unit 105a, the joint position estimation unit 108, the joint position task learning unit 109,The task determination unit 107a, the object recognition unit 1071, the dynamic body recognition unit 1072, and the joint position task estimation unit 1073 in the task analysis device 1A are provided by a server. Furthermore, each function can be realized by the task analysis devices 1 and 1A using a cloud-based virtual server function or the like.
[0136] Furthermore, the task analysis devices 1 and 1A may be a distributed processing system in which each function of the task analysis devices 1 and 1A is distributed among a plurality of servers as needed. <Dritte Variante>
[0137] As another example, in the above-described embodiments, the task analysis device 1A does not have the object recognition annotation suggestion unit 106, but may have the object recognition annotation suggestion unit 106.
[0138] As a result, in a case where the accuracy of task determination is insufficient, the task analysis device 1A can automatically suggest an image in a moving image with which the accuracy of task determination can be increased when annotating.
[0139] Note that each function included in the task analysis devices 1 and 1A according to the first embodiment and the second embodiment can be implemented by hardware, software, or a combination thereof. Software implementation means that it is implemented by a computer reading and executing a program.
[0140] A program can be stored on various types of non-transferable, computer-readable media and delivered to a computer. A non-transitory, computer-readable medium includes various types of tangible storage media. An example of a non-transitory computer-readable medium is a magnetic recording medium (e.g., a floppy disk, magnetic tape, or hard disk drive), a magneto-optical recording medium (e.g., a magneto-optical disk), a CD-ROM (read-only memory), CD-R, CD-R / W, and a semiconductor memory (e.g., a mask ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash ROM, or a RAM). In addition, a program can be delivered to a computer through various types of transitory computer-readable media. An example of a transitory computer-readable medium is an electrical signal, an optical signal, or electromagnetic waves.A transitory computer-readable medium can deliver a program to a computer over a wired communications channel, such as an electrical wire or an optical fiber, or over a wireless communications channel.
[0141] It should be noted that steps expressing a program recorded on a recording medium naturally include processing in chronological order according to the order of these steps, but also include processing executed in parallel or individually without the need for processing in chronological order.
[0142] To rephrase the above, a task analysis device according to the present disclosure may have various embodiments having configurations such as the following.
[0143] (1) The task analysis device 1 according to the present disclosure is a task analysis device for analyzing a task by a worker, and is provided with: the task label assignment unit 102 that assigns a task label indicating the worker's task to video data containing the worker's task; the object recognition annotation unit 103 that, with respect to the video data to which the task label has been assigned, makes an annotation for an object related to the worker's task; the object recognition learning unit 104 that generates an object recognition model for performing object recognition from video data concerning the object for which the annotation was made by the object recognition annotation unit 103; the object recognition unit 1071 that uses the object recognition model to detect the object from the video data;the task determination parameter calculation unit 105, which performs task determination on the video data assigned the task label and calculates a determination criterion for minimizing an error with respect to the assigned task label; and the task determination unit 107, which uses the object recognition model and the determination criterion to determine a task by the worker in newly input video data.
[0144] With the aid of this task analysis device 1, it is possible to automatically set a determination criterion so that a task can be accurately determined.
[0145] (2) The task analysis device 1 according to (1) may include the object recognition annotation suggestion unit 106 that performs task determination on the video data to which the task label has been assigned using the determination criterion calculated by the task determination parameter calculation unit 105, and suggests a frame for creating an annotation based on a determination result from the task determination.
[0146] As a result, in a case where the accuracy of the task determination is insufficient, the task analysis device 1 can automatically suggest a frame in a moving image with which the accuracy of the task determination can be increased when making an annotation.
[0147] (3) The task analysis device 1A according to (1) or (2) may include: the joint position estimation unit 108 that estimates joint position information related to joint positions for the worker; the joint position task learning unit 109 that generates a joint position task estimation model for estimating the task by the worker based on the joint position information estimated by the joint position estimation unit 108 and information related to the task label assigned by the task label assignment unit 102;and the joint position task estimation unit 1073 estimates a task from the joint position information based on the joint position task estimation model generated by the joint position task learning unit 109; the task determination parameter calculation unit 105a calculates the determination criterion based on a value related to the accuracy of object detection in the task determination that used the object detection model and a task classification probability estimated from joint positions in the task determination that used the joint position task estimation model so as to minimize an error related to the task designation; and the task determination unit 107a determines a task by the worker in newly input video data using the object detection model, the joint position task estimation model, and the determination criterion.
[0148] As a result, the task analysis device 1A can achieve an effect similar to that of (1).
[0149] (4) The task analysis device 1 or 1A according to any one of (1) to (3) may further include: the dynamic body recognition unit 1072 that recognizes a dynamic body in the newly input video data, the task determination unit 107 or 107a that determines whether the worker's task continues based on a time interval in which the dynamic body recognition unit 1072 has recognized the dynamic body.
[0150] As a result, the task analysis device 1 or 1A can determine a task by a worker with better accuracy.
[0151] (5) In the task analysis device 1 or 1A according to any one of (1) to (4), the determination criterion may include at least a threshold value for a value related to the accuracy of object recognition and a period of time for which a task using a tool (object) can be estimated to continue from the time the tool (object) is recognized.
[0152] As a result, the task analysis device 1 or 1A can accurately determine a task by a worker even if a tool (object) is not recognized. EXPLANATION OF REFERENCE NUMBERS 1, 1A Task analysis device 2 cameras 10, 10a control unit 101 Task Registration Unit 102 Task Label Assignment Unit 103 Object recognition annotation unit 104 Object Recognition Learning Unit 105, 105a Task determination parameter calculation unit 106 Object recognition annotation suggestion unit 107, 107a Task determination unit 1071 Object recognition unit 1072 Dynamic Body Detection Unit 1073 Joint position task estimation unit 108 Joint position estimation unit 109 Joint Position Task Learning Unit 20 storage units 201 Video data storage unit 202 Task registration storage unit 203 Input data storage unit QUOTES CONTAINED IN THE DESCRIPTION
[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature
[0000] 2021-67981
[0004] WO2017 / 222070
[0004] Cited non-patent literature
[0000] SUGANO, Kousuke, OKU, Kenta, KAWAGOE, Kyoji, “Motion detection and classification method from multidimensional time series data”, DEIM Forum 2016 G4-5
[0100] UEZONO, Shouhei, ONO, Satoshi, “Feature extraction using LSTM Autoencoder for multimodal sequential data”, JSAI Technical Report, SIG-KBS-B802-1, 2018
[0100]
Claims
[1] A task analysis device for analyzing a task by a worker, the task analysis device comprising: a task label assigning unit configured to assign a task label indicating the worker's task to video data containing the worker's task; an object detection annotation unit configured to annotate an object related to the worker's task with respect to the video data assigned the task label; an object recognition learning unit configured to generate an object recognition model for performing object recognition from video data regarding the object for which the annotation was made by the object recognition annotation unit; an object detection unit configured to use the object detection model to detect the object from the video data; a task determination parameter calculation unit configured to perform task determination on the video data to which the task label has been assigned and to calculate a determination criterion for minimizing an error with respect to the assigned task label; and a task determination unit configured to use the object detection model and the determination criterion to determine a task by the worker in newly input video data. [2] The task analysis device according to claim 1, comprising an object recognition annotation suggestion unit configured to perform task determination on the video data to which the task label has been assigned using the determination criterion calculated by the task determination parameter calculation unit, and suggest a frame for creating an annotation based on a determination result from the task determination. [3] The task analysis device according to claim 1 or 2, comprising: a joint position estimation unit configured to estimate joint position information concerning joint positions of the worker; a joint position task learning unit configured to generate a joint position task estimation model for estimating the task by the worker based on the information about joint positions estimated by the joint position estimation unit and information about the task label assigned by the task label assignment unit; and a joint position task estimation unit configured to estimate a task from the information on joint positions based on the joint position task estimation model generated by the joint position task learning unit, wherein based on a value concerning the accuracy of object detection in the task determination using the object recognition model and a task classification probability estimated from joint positions in the task determination using the joint position task estimation model, the task determination parameter calculation unit calculates the determination criterion so as to minimize an error with respect to the task label, and the task determination unit determines a task by the worker in newly input video data using the object detection model, the joint position task estimation model, and the determination criterion. [4] The task analysis device according to any one of claims 1 to 3, further comprising: a dynamic body detection unit configured to detect a dynamic body in the newly input video data, wherein the task determination unit determines whether to continue the worker's task based on a time interval in which the dynamic body detection unit has detected the dynamic body. [5] The task analysis device according to any one of claims 1 to 4, wherein the determination criterion includes at least a threshold value for a value related to the accuracy of object recognition and a period of time for which a task using the object can be estimated to continue when the object is recognized.
Citation Information
Patent Citations
Work analysis device and work analysis method
JP2021067981A
Work analysis device, work analysis method, and computer-readable recording medium
WO2017222070A1
JP002021067981A
