Work evaluation device, work evaluation method, and program

Abstract

This work evaluation device calculates, for each of a plurality of frame columns included in video data, a first evaluation value based on the change over time in the size of an image region of a tool detected from the frame column, a second evaluation value based on the change over time in the degree of bending of a finger detected from the frame column, or both the first evaluation value and the second evaluation value, thereby calculating an individual work evaluation value indicating the level of evaluation of a work which is being imaged in the frame column. The video data includes a plurality of frame columns each of which is composed of a plurality of continuous video frames that belong to the same class. Adjacent frame columns belong to different classes. A class indicates a type of work.

Description

Work evaluation device, work evaluation method, and program

[0001] The present disclosure relates to an activity evaluation device, an activity evaluation method, and a program.

[0002] Systems for evaluating behavior using video have been developed. For example, Patent Document 1 discloses a technology for using images to determine whether a virtual task performed in a sterile work apparatus is appropriate. Patent Document 1 determines whether the position of a worker's hand, finger, or forearm is too close to a predetermined location, whether the speed of the movement is within a standard speed, and whether the movement or position is within a standard. Here, the indicators for representing the movement of the hand or the like include the movement speed, movement direction, and movement vector.

[0003] Japanese Patent Application Laid-Open No. 2019-191348

[0004] In Patent Document 1, tasks are evaluated using the indices "the relationship between the position of the hand, finger, or forearm and a reference" and "the relationship between the speed, direction, and movement vector of the hand, finger, or forearm and a reference." Therefore, Patent Document 1 does not mention evaluating tasks using indices other than these indices. The present disclosure has been made in consideration of this problem, and one of its purposes is to provide a new technology for evaluating tasks using video.

[0005] The task evaluation device of the present disclosure includes an acquisition means for acquiring video data including a plurality of frame sequences each composed of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes, and a calculation means for calculating, for each of the frame sequences, a first evaluation value based on a change over time in the size of an image area of ​​a tool detected from the frame sequence, a second evaluation value based on a change over time in the degree of bending of a finger detected from the frame sequence, or both the first evaluation value and the second evaluation value, thereby calculating an individual task evaluation value representing a level of evaluation of the task captured in the frame sequence. The class represents a type of task.

[0006] The disclosed task evaluation method is executed by a computer. The method includes an acquisition step of acquiring video data including a plurality of frame sequences each composed of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes, and a calculation step of calculating, for each of the frame sequences, a first evaluation value based on a time change in the size of an image area of ​​a tool detected in the frame sequence, a second evaluation value based on a time change in the degree of bending of a finger detected in the frame sequence, or both the first evaluation value and the second evaluation value, thereby calculating an individual task evaluation value representing a level of evaluation of the task captured in the frame sequence. The class represents a type of task.

[0007] The program of the present disclosure causes a computer to execute the task evaluation method of the present disclosure.

[0008] According to the present disclosure, a new technique for evaluating work using video is provided.

[0009] FIG. 1 is a diagram illustrating video data handled by the activity evaluation device. FIG. 2 is a diagram illustrating an overview of the operation of the activity evaluation device. FIG. 3 is a block diagram illustrating the functional configuration of the activity evaluation device. FIG. 4 is a block diagram illustrating the hardware configuration of a computer that realizes the activity evaluation device. FIG. 5 is a flowchart illustrating the flow of processing executed by the activity evaluation device. FIG. 6 is a diagram illustrating class membership information. FIG. 7 is a first diagram illustrating class definition information. FIG. 8 is a second diagram illustrating class membership information.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] 1 is a diagram illustrating video data 10 handled by the task evaluation device. The video data 10 is made up of a plurality of time-series video frames 12. In other words, the video data 10 is a frame sequence in which a plurality of video frames 12 are arranged in time-series order (in ascending order of frame numbers).

[0012] Each video frame 12 belongs to one of a plurality of classes. The video data 10 includes a plurality of frame sequences 20. A frame sequence 20 is a frame sequence made up of a plurality of consecutive video frames 12 that belong to the same class. For example, the video data 10 in FIG. 1 includes, in this order, a frame sequence 20-1 made up of a plurality of video frames 12 that belong to class C1, a frame sequence 20-2 made up of a plurality of video frames 12 that belong to class C2, and a frame sequence 20-3 made up of a plurality of video frames 12 that belong to class C3.

[0013] Hereinafter, a frame sequence 20 consisting of a plurality of video frames 12 belonging to class C will also be referred to as a "frame sequence 20 belonging to class C." In this case, it will also be expressed as "the frame sequence 20 belongs to class C." Furthermore, the class to which the frame sequence 20 belongs will also be referred to as the "class of the frame sequence 20." If the frame sequence 20 belongs to class C, the class of the frame sequence 20 is C.

[0014] Here, the video data 10 includes at least two frame sequences 20 that belong to different classes. Note that the video data 10 may also include two or more frame sequences 20 that belong to the same class. For example, as in the example of Figure 1, in a case where the video data 10 includes three frame sequences 20, frame sequence 20-1 to frame sequence 20-3, frame sequence 20-1 and frame sequence 20-3 may belong to class C1, and frame sequence 20-2 may belong to class C2.

[0015] The video data 10 includes a scene in which a worker (a person performing a task) is performing a task. The class of the frame sequence 20 indicates the type of task being performed by the worker in the scene captured in the frame sequence 20.

[0016] For example, suppose a video camera captures the scenes of workers performing tasks of type A1, type A2, and type A3. The video data obtained by the video capture is treated as video data 10. In this case, A1, A2, and A3 are each treated as a class. The video data 10 is divided into three frame sequences 20: a frame sequence 20 capturing the state of task A1, a frame sequence 20 capturing the state of task A2, and a frame sequence 20 capturing the state of task A3.

[0017] Fig. 2 is a diagram illustrating an example of an outline of the operation of the work evaluation device 2000. Here, Fig. 2 is a diagram for facilitating understanding of the outline of the work evaluation device 2000, and the operation of the work evaluation device 2000 is not limited to that shown in Fig. 2.

[0018] The task evaluation device 2000 analyzes the frame sequence 20 to calculate an evaluation value (hereinafter, referred to as an individual task evaluation value) that indicates the appropriateness of the task captured in the frame sequence 20. The task evaluation device 2000 calculates at least a first evaluation value or a second evaluation value to calculate the individual task evaluation value. The individual task evaluation value is a value based on the first evaluation value, the second evaluation value, or both.

[0019] The first evaluation value is calculated based on a change over time in the size of an image area representing a tool used for work in the frame sequence 20. Hereinafter, the tool used for work will be referred to as a work tool. Also, the image area representing the work tool will be referred to as a tool area.

[0020] The second evaluation value is calculated based on the change over time in the degree of bending of the specific finger of the worker in the frame sequence 20. Hereinafter, the specific finger of the worker will also be referred to as a finger of interest.

[0021] <Example of Effects> According to the task evaluation device 2000, an individual task evaluation value indicating the level of evaluation of the task is calculated for each of a plurality of frame sequences 20 included in the video data 10. The individual task evaluation value is a value based on the first evaluation value, the second evaluation value, or both. The first evaluation value is calculated based on the change over time in the size of an image area representing a tool used in the task in the frame sequence 20. The second evaluation value is calculated based on the change over time in the degree of bending of a specific finger of the worker in the frame sequence 20.

[0022] From the above, the task evaluation device 2000 calculates a value representing the task evaluation using a new index not disclosed in Patent Document 1, namely, the change over time in the size of an image region representing the tool used in the task, or the change over time in the size of an image region representing the tool used in the task. Thus, the task evaluation device 2000 provides a new technique for evaluating tasks using video.

[0023] The reason why the first evaluation value and the second evaluation value are used will be explained below.

[0024] While a worker is performing a task, the appearance of the work tool being used by the worker may change. For example, the appearance of the work tool changes when the worker changes the posture of the work tool or the way the work tool is held. Therefore, the change in the appearance of the work tool over time represents the state of the work performed by the worker. Furthermore, as the appearance of the work tool changes, the size of the tool area, which is the image area representing the work tool, changes in the video frame 12.

[0025] Therefore, the work evaluation device 2000 calculates a first evaluation value based on the change in the tool area over time. By using the first evaluation value, it is possible to evaluate whether the worker is performing the work appropriately. In particular, the change in the appearance of the work tool over time represents how the work tool is being used by the worker. Therefore, by using the first evaluation value, it is possible to evaluate whether the worker is using the work tool correctly.

[0026] Furthermore, the degree of bending of the worker's fingers may change while the worker is performing a task. Therefore, the change over time in the degree of bending of the worker's target finger represents the state of the worker's task.

[0027] Therefore, the task evaluation device 2000 calculates a second evaluation value based on the change over time in the degree of bending of the target finger. By using the second evaluation value, it is possible to evaluate whether the worker is performing the task appropriately. In particular, the change over time in the degree of bending of the target finger represents the manner in which the worker is moving the finger. Therefore, by using the second evaluation value, it is possible to evaluate whether the worker is moving the finger correctly.

[0028] The task evaluation device 2000 of this embodiment will be described in more detail below.

[0029] <Example of Functional Configuration> Fig. 3 is a block diagram illustrating an example of the functional configuration of the task evaluation device 2000. The task evaluation device 2000 has an acquisition unit 2020 and a calculation unit 2040. The acquisition unit 2020 acquires video data 10. The calculation unit 2040 calculates an individual task evaluation value for each frame sequence 20 included in the video data 10. The calculation of the individual task evaluation value includes calculation of a first evaluation value, calculation of a second evaluation value, or both.

[0030] <Example of Hardware Configuration> Each functional component of the work evaluation device 2000 may be realized by hardware that realizes the functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the work evaluation device 2000 is realized by a combination of hardware and software will be further described.

[0031] 4 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the work evaluation device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the work evaluation device 2000, or may be a general-purpose computer.

[0032] For example, by installing a predetermined application on the computer 1000, the computer 1000 realizes each function of the work evaluation device 2000. The application is configured as a program for realizing each functional component of the work evaluation device 2000. The program can be acquired by any method. For example, the program can be acquired from a storage medium on which the program is stored. The storage medium on which the program is stored can be any storage medium such as a DVD (Digital Versatile Disk) or a USB (Universal Serial Bus) memory. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0033] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0034] The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0035] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0036] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0037] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the work evaluation device 2000. The processor 1040 reads this program into the memory 1060 and executes it, thereby realizing each functional component of the work evaluation device 2000.

[0038] The task evaluation device 2000 may be realized by one computer 1000, or may be realized by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from one another.

[0039] 5 is a flowchart illustrating the flow of processing executed by the task evaluation device 2000. The acquisition unit 2020 acquires video data 10 (S102). The calculation unit 2040 calculates an individual task evaluation value for each frame sequence 20 included in the video data 10.

[0040] <Acquisition of Video Data 10: S102> The acquisition unit 2020 acquires the video data 10. Here, various methods can be used to acquire the frame sequence to be processed. For example, the video data 10 is stored in advance in an arbitrary storage device in a format that allows it to be acquired by the work evaluation device 2000. In this case, the acquisition unit 2020 acquires the video data 10 by reading the video data 10 from the storage device.

[0041] Alternatively, for example, the acquisition unit 2020 acquires the video data 10 by receiving the video data 10 transmitted from another device. The device that transmits the video data 10 is, for example, the device that generated the video data 10. If the video data 10 is video data, for example, the acquisition unit 2020 acquires the video data 10 from the video camera that generated the video data 10.

[0042] <<Information for Identifying the Class of Video Frames 12>> The task evaluation device 2000 extracts each frame sequence 20 from the video data 10 in order to calculate an individual task evaluation value for each frame sequence 20. There are various methods for extracting each frame sequence 20 from the video data 10.

[0043] For example, the task evaluation device 2000 acquires information indicating the class to which each video frame 12 or each frame sequence 20 belongs (hereinafter referred to as class membership information).

[0044] For example, the class membership information indicates, for each video frame 12 included in the video data 10, a correspondence between its identification information (e.g., frame number) and the identification information of the class to which the video frame 12 belongs. Alternatively, for each frame sequence 20 included in the video data 10, the class membership information indicates, for example, the class to which the frame sequence 20 belongs and the identification information of either or both of the first and last video frames 12 of the frame sequence 20.

[0045] 6 is a diagram illustrating class membership information. A table 200 indicates, for each video frame 12, the class to which that video frame 12 belongs. More specifically, the table 200 indicates, in association with the identification information of the video frame 12 (frame identification information 202), the identification information of the class to which that video frame 12 belongs (class identification information 204).

[0046] On the other hand, the table 300 indicates, for each frame sequence 20, the class to which that frame sequence 20 belongs. More specifically, for each frame sequence 20, the table 300 indicates the identification information of the class to which that frame sequence 20 belongs (class identification information 306) in association with a combination of the identification information of the first video frame 12 (first frame identification information 302) and the identification information of the last video frame 12 (last frame identification information 304).

[0047] The class membership information may be information that is integrated with the video data 10, or may be information that is separate from the video data 10. In the former case, for example, identification information of the class to which each video frame 12 included in the video data 10 belongs is added as metadata. When the video data 10 and the class membership information are configured separately, for example, the acquisition unit 2020 further acquires the class membership information for the video data 10 in addition to the video data 10. The method of acquiring the class membership information is the same as the method of acquiring the video data 10.

[0048] <Calculation of Individual Task Evaluation Value: S104> The calculation unit 2040 calculates an individual task evaluation value for each frame sequence 20. As described above, the calculation of the individual task evaluation value includes calculation of a first evaluation value, calculation of a second evaluation value, or both. Below, the methods for calculating the first evaluation value and the second evaluation value will be described.

[0049] <<Method of Calculating First Evaluation Value>> The first evaluation value of the task captured in the frame sequence 20 is calculated based on the change over time in the size of the tool region in the frame sequence 20. Therefore, the calculation unit 2040 detects a tool region from each video frame 12 included in the frame sequence 20. Furthermore, the calculation unit 2040 generates time-series data of the size of the tool region detected from each video frame 12. Hereinafter, this time-series data will be referred to as first real time-series data.

[0050] For example, suppose the frame sequence 20 is composed of a video frame F1 generated at time t1, a video frame F2 generated at time t2, and a video frame F3 generated at time t3. Furthermore, suppose the tool regions in the video frames F1 to F3 have sizes S1, S2, and S3, respectively. In this case, the calculation unit 2040 generates first real time series data {(t1, S1), (t2, S2), (t3, S3)}.

[0051] There are various methods for detecting a tool region from a video frame 12. For example, the calculation unit 2040 detects a tool region from a video frame 12 by performing object recognition processing on the video frame 12. For example, assume that the only objects detected from the video frame 12 are the worker's body (hands, etc.) and a work tool. In this case, the calculation unit 2040 recognizes, among the objects detected from the video frame 12, the objects other than the worker's body as work tools.

[0052] On the other hand, it is assumed that the video frames 12 may also include objects other than the worker's body and work tools. In this case, for example, the objects used as work tools are determined in advance. Information indicating the objects used as work tools is referred to as tool information. The tool information is, for example, stored in advance in a storage unit accessible by the work evaluation device 2000. The calculation unit 2040 acquires the tool information and detects, from among the objects detected in the video frames 12, the objects indicated in the tool information as work tools.

[0053] Objects used as work tools may be defined for each type of work (i.e., class). Hereinafter, information indicating various information related to each class will be referred to as class definition information. The class definition information indicates objects used as work tools in association with the class. Of the objects detected from the video frames 12, the calculation unit 2040 detects, as work tools, objects associated with the classes of the frame sequence 20.

[0054] The tool region is, for example, a region represented by a circumscribing rectangle of the tool. However, the tool region is not limited to a circumscribing rectangle of the tool, and can be any region representing the tool.

[0055] The calculation unit 2040 acquires time-series data (hereinafter referred to as first reference time-series data) that represents a reference for time-series changes in the size of the tool area. The first reference time-series data represents time-series changes in the size of the tool area when the task is performed ideally. The first reference time-series data is stored in advance in, for example, a storage unit accessible from the task evaluation device 2000.

[0056] The first reference time series data is, for example, predetermined for each class. For example, the first reference time series data is indicated in the class definition information. The calculation unit 2040 acquires the first reference time series data associated with the class of the frame sequence 20 from the class definition information.

[0057] FIG. 7 is a first diagram illustrating an example of class definition information. In FIG. 7, class definition information 400 indicates a task type 404, tool identification information 406, and first reference time series data 408, all associated with class identification information 402. The task type 404 indicates the name of the type of task associated with the class. The tool identification information 406 indicates identification information for identifying the work tool associated with the class. For example, the tool identification information 406 indicates image features of an object used as a work tool. In this case, the calculation unit 2040 detects, as a work tool, an object detected from the video frame 12 that has features matching features associated with a class in the frame sequence 20. The first reference time series data 408 indicates the first reference time series data associated with the class.

[0058] It can be said that the more similar the first actual time series data is to the first reference time series data, the more appropriately the worker is performing the work. Therefore, the calculation unit 2040 calculates the first evaluation value by comparing the first actual time series data with the first reference time series data. For example, the calculation unit 2040 calculates the similarity between the first actual time series data and the first reference time series data, and treats the calculated similarity as the first evaluation value.

[0059] There are various methods for calculating the similarity between two pieces of time series data. For example, the calculation unit 2040 calculates the distance between the first actual time series data and the first reference time series data using dynamic time warping (DTW). The calculation unit 2040 then calculates the first evaluation value based on the calculated distance. Alternatively, for example, the calculation unit 2040 may calculate the cross-correlation between the first actual time series data and the first reference time series data using a cross-correlation function (CCF) and treat the calculated cross-correlation as the first evaluation value.

[0060] The first evaluation value is expressed as a value that increases as the similarity between the first actual time series data and the first reference time series data increases, for example. In this case, the first evaluation value may be, for example, the inverse of the distance between the first actual time series data and the first reference time series data, or the cross-correlation between the first actual time series data and the first reference time series data.

[0061] The first evaluation value may be expressed as a value that decreases as the similarity between the first actual time series data and the first reference time series data increases. In this case, for example, the first evaluation value may be a distance between the first actual time series data and the first reference time series data.

[0062] Here, the first actual time series data and the first reference time series data may have different scales with respect to the size of the tool area, for example, because the distance between the camera and the worker or the zoom setting of the camera may be different between the shooting performed to generate the first reference time series data and the shooting performed to generate the video data 10.

[0063] Therefore, the calculation unit 2040 may perform a process to match the scale of the tool region size between the first real time series data and the first reference time series data. For example, the calculation unit 2040 calculates the average value of the tool region size for both the first real time series data and the first reference time series data. Then, the calculation unit 2040 multiplies the first real time series data or the first reference time series data by a correction coefficient for correcting the scale of the tool region size so that these average values ​​match.

[0064] For example, let Sr be the average value of the tool region in the first real time series data, and Sc be the average value of the tool region in the first reference time series data. In this case, for example, the calculation unit 2040 corrects the size of the tool region in the first real time series data by multiplying it by a correction coefficient Sc / Sr. Alternatively, the calculation unit 2040 may correct the size of the tool region in the first reference time series data by multiplying it by the correction coefficient Sr / Sc.

[0065] The first actual time series data and the first reference time series data may have different lengths along the time axis. Therefore, the calculation unit 2040 may perform a correction to align the lengths along the time axis of the first actual time series data and the first reference time series data. For example, the calculation unit 2040 may thin out data from the longer time series data to equalize the lengths of the first actual time series data and the first reference time series data. Alternatively, for example, the calculation unit 2040 may interpolate data in the shorter time series data to equalize the lengths of the first actual time series data and the first reference time series data.

[0066] <<Method of Calculating Second Evaluation Value>> The second evaluation value of the task captured in the frame sequence 20 is calculated based on the change over time in the degree of bending of the target finger in the frame sequence 20. To achieve this, the calculation unit 2040 calculates a value representing the degree of bending of the target finger for each video frame 12, and generates time-series data of the calculated values. Hereinafter, this time-series data will be referred to as second real time-series data.

[0067] The degree of bending of the target is expressed, for example, by the distance between the tip of the target finger and the wrist of the worker, because the distance between the tip of the finger and the wrist changes depending on the degree of bending of the finger.

[0068] For example, the degree of bending of the target finger is calculated using the following formula (1). B[i] represents the degree of bending of the target finger. L[i] represents the distance between the tip of the target finger and the wrist in the i-th video frame 12 of the frame sequence 20. Lm represents the distance between the tip of the target finger when it is not bent and the wrist. In other words, Lm represents the maximum distance between the tip of the target finger and the wrist.

[0069] The calculation unit 2040 generates second real time series data by calculating B[i] for each video frame 12 included in the frame sequence 20. For example, assume that the frame sequence 20 is composed of a video frame F1 generated at time t1, a video frame F2 generated at time t2, and a video frame F3 generated at time t3. In this case, the calculation unit 2040 generates second real time series data of {(t1, B[1]), (t2, B[2]), (t3, B[3])}.

[0070] To calculate the degree of bending B[i] of the finger of interest for each video frame 12, the calculation unit 2040 detects the tip of the finger of interest and the wrist from each video frame 12. For example, the calculation unit 2040 detects a plurality of joint points (points representing the positions of the joints) of the worker's hand by performing skeletal detection on the video frame 12. Furthermore, the calculation unit 2040 identifies the joint point representing the position of the tip of the finger of interest and the joint point representing the position of the worker's wrist. Then, the calculation unit 2040 calculates the distance between the joint point representing the position of the tip of the finger of interest and the joint point representing the position of the wrist. As a result, the distance L[i] between the tip of the finger of interest and the wrist is calculated.

[0071] There are various methods for calculating the maximum distance Lm between the tip of the finger of interest and the wrist. For example, the calculation unit 2040 detects a video frame 12 in which the target finger is not bent from among multiple video frames 12. Then, the calculation unit 2040 calculates the distance between the tip of the target finger and the wrist in the detected video frame 12 as the maximum distance between the tip of the target finger and the wrist.

[0072] Alternatively, for example, the calculation unit 2040 may calculate the maximum distance between the tip of the target finger and the wrist by calculating the sum of the distances between the joints of the target finger. For example, suppose two joint points of the target finger are detected: one at the tip of the target finger and one at the base of the target finger. In this case, the calculation unit 2040 calculates the maximum distance between the tip of the target finger and the wrist by adding the distance between the wrist joint point and the base of the target finger and the distance between the base of the target finger and the tip of the target finger.

[0073] Here, the maximum distance between the tip of the finger of interest and the wrist can be a common value across multiple frame sequences 20. Therefore, for example, the calculation unit 2040 calculates the maximum distance between the tip of the finger of interest and the wrist for only the first frame sequence 20. Then, the calculation unit 2040 uses the calculated maximum distance between the tip of the finger of interest and the wrist when calculating B[i] for the other frame sequences 20.

[0074] In addition to calculating the second actual time-series data, the calculation unit 2040 acquires time-series data (hereinafter, referred to as second reference time-series data) that is defined as a reference for time-series changes in the degree of bending of the target finger. The second reference time-series data represents time-series changes in the degree of bending of the target finger when the task is performed ideally. The second reference time-series data is stored in advance in, for example, a storage unit accessible from the task evaluation device 2000.

[0075] The second reference time series data is, for example, predetermined for each class. In this case, the second reference time series data is included in class definition information 400. FIG. 8 is a second diagram illustrating class definition information. In FIG. 8, the class definition information 400 indicates an operation type 404 and second reference time series data 410 in association with class identification information 402. The second reference time series data 410 indicates the second reference time series data associated with the class. Note that when both the first evaluation value and the second evaluation value are calculated, the class definition information 400 in FIG. 8 further indicates tool identification information 406 and first reference time series data 408.

[0076] The more similar the second actual time series data is to the second reference time series data, the more appropriately the worker is performing the work. Therefore, the calculation unit 2040 calculates the second evaluation value by comparing the second actual time series data with the second reference time series data. For example, the calculation unit 2040 calculates the similarity between the second actual time series data and the second reference time series data, and treats the calculated similarity as the second evaluation value. Here, the similarity between the two time series data can be calculated using, for example, the various information methods described above (such as methods using DTW or CCF).

[0077] The second evaluation value is expressed as, for example, a value that increases as the similarity between the second actual time series data and the second reference time series data increases. In this case, the second evaluation value may be, for example, the reciprocal of the distance between the second actual time series data and the second reference time series data, or the cross-correlation between the second actual time series data and the second reference time series data.

[0078] The second evaluation value may be expressed as a value that decreases as the similarity between the second actual time series data and the second reference time series data increases. In this case, for example, the first evaluation value may be a distance between the first actual time series data and the first reference time series data.

[0079] Here, the calculation unit 2040 may perform a correction to align the lengths of the second actual time series data and the second reference time series data in the time axis direction. The method of correcting to align the lengths of the two time series data in the time axis direction is as described above.

[0080] <<Regarding the Case Where Both the First Evaluation Value and the Second Evaluation Value are Calculated>> When both the first evaluation value and the second evaluation value are calculated, the calculation unit 2040 may calculate, as the individual task evaluation value, a statistical value of the first evaluation value and the second evaluation value for each frame sequence 20. The statistical value may be, for example, a simple average value or a weighted average value.

[0081] For example, when the individual task evaluation value is expressed as a weighted average of the first evaluation value and the second evaluation value, the individual task evaluation value can be calculated using the following formula (2). i represents the identifier of the frame sequence 20. I[i] represents the individual task evaluation value calculated for the frame sequence 20 with identifier i (hereinafter referred to as frame sequence i). X[i] and Y[i] represent the first evaluation value and the second evaluation value calculated for frame sequence i, respectively. a and b represent the weights assigned to the first evaluation value and the second evaluation value, respectively.

[0082] <Calculation of Other Evaluation Values> The calculation unit 2040 may further calculate an evaluation value for the entire video data 10 using the individual task evaluation values ​​calculated for each frame sequence 20. The evaluation value for the entire video data 10 represents a comprehensive evaluation of multiple tasks performed by the worker. Hereinafter, the evaluation value for the entire video data 10 will be referred to as a comprehensive evaluation value.

[0083] For example, the calculation unit 2040 calculates, as the overall evaluation value, a statistical value (such as a simple average or a weighted average) of the individual task evaluation values ​​calculated for each of the multiple frame sequences 20 included in the video data 10. The weights assigned to the individual task evaluation values ​​are, for example, predetermined for each class.

[0084] The statistical value of the individual task evaluation value can be calculated using, for example, the following equation (3). SI represents the statistical value of the individual task evaluation values ​​calculated for the entire video data 10. More specifically, in equation (3), SI is the weighted average of the individual task evaluation values. C[i] represents the class corresponding to frame sequence i. W(x) represents the weight assigned to class x.

[0085] The weights assigned to the classes are indicated, for example, in the class definition information 400. In this case, the calculation unit 2040 obtains, for each frame sequence 20, the weight W(C[i]) corresponding to the class C[i] of the frame sequence 20 from the class definition information 400.

[0086] The calculation unit 2040 may further calculate an evaluation value representing the appropriateness of the order of tasks (hereinafter referred to as the order evaluation value), an evaluation value representing the appropriateness of the length of task time (hereinafter referred to as the time evaluation value), an evaluation value representing the appropriateness of a repetitive action (hereinafter referred to as the repetitive action evaluation value), etc. The calculation method for each evaluation value will be described later.

[0087] For example, the calculation unit 2040 calculates statistics for the individual task evaluation values, the sequence evaluation values, the time evaluation values, and the repetitive action evaluation values, and treats the calculated statistics as the overall evaluation value. In this case, for example, the overall evaluation value is calculated using the following formula (4). E represents the overall evaluation value. O, T, and R represent the order evaluation value, time evaluation value, and repetitive action evaluation value, respectively. c, d, e, and f are the weights given to the statistics of the individual task evaluation value, the order evaluation value, the time evaluation value, and the repetitive action evaluation value, respectively.

[0088] Hereinafter, the calculation methods for the sequence evaluation value, the time evaluation value, and the repetitive movement evaluation value will be described.

[0089] <Method of Calculating Order Evaluation Value> The order evaluation value represents the degree to which the order of tasks represented by the video data 10 is close to the correct order of tasks. For example, the calculation unit 2040 calculates, as the order evaluation value, a value representing the degree of similarity between a permutation representing the order of tasks represented by the video data 10 and a permutation representing the correct tasks.

[0090] For example, suppose that video data 10 includes, in this order, a frame sequence 20 belonging to class C1, a frame sequence 20 belonging to class C2, a frame sequence 20 belonging to class C3, and a frame sequence 20 belonging to class C4. In this case, the permutation representing the order of tasks represented by the video data 10 is (C1, C2, C3, C4).

[0091] Also, suppose the correct order of operations is class C2 operations, class C3 operations, class C4 operations, and class C1 operations. In this case, the permutation that represents the correct order of operations is (C2, C3, C4, C1).

[0092] The degree of similarity between two permutations can be expressed, for example, using the edit distance between the two permutations. Here, the smaller the edit distance between the permutation representing the order of tasks represented by the video data 10 and the permutation representing the correct order of tasks, the more similar the order of tasks actually performed by the worker is to the correct order of tasks. Therefore, the smaller the edit distance, the more appropriate the order of tasks performed by the worker is. Therefore, for example, the calculation unit 2040 calculates, as the order evaluation value, a value that increases as the edit distance decreases, such as the reciprocal of the edit distance.

[0093] However, the order evaluation value may be a value that decreases as the order of the work performed by the workers becomes more appropriate. In this case, for example, the edit distance described above is used as the order evaluation value.

[0094] To calculate the order evaluation value, the calculation unit 2040 acquires data representing the correct task order (hereinafter, referred to as reference order data). The reference order data is stored in advance in a storage unit accessible from the task evaluation device 2000, for example.

[0095] <Method of Calculating Time Evaluation Value> The time evaluation value is an evaluation value that represents the appropriateness of the length of task time. For example, for each frame sequence 20 included in the video data 10, the calculation unit 2040 determines whether the length of the task time represented by that frame sequence 20 is appropriate, and calculates the time evaluation value based on the result of the determination.

[0096] For example, the time evaluation value is expressed as the number of frame sequences 20 for which the length of the work time is determined to be appropriate, or the number of frame sequences 20 for which the length of the work time is determined to be inappropriate. In the former case, the more appropriate the length of the work time, the larger the time evaluation value. On the other hand, in the latter case, the more appropriate the length of the work time, the smaller the time evaluation value.

[0097] As another example, the time evaluation value may be expressed as the ratio of the number of frame sequences 20 whose working time length is determined to be appropriate to the total number of frame sequences 20 included in the video data 10. In this case, the more appropriate the working time length, the larger the time evaluation value. As another example, the time evaluation value may be expressed as the ratio of the number of frame sequences 20 whose working time length is determined to be inappropriate to the total number of frame sequences 20 included in the video data 10. In this case, the more appropriate the working time length, the smaller the time evaluation value.

[0098] To determine whether the length of work time is appropriate, the calculation unit 2040 calculates the allowable length of work time (hereinafter, the allowable work time value). For example, an appropriate length of work time (hereinafter, the reference work time value) is determined in advance for each class. The calculation unit 2040 acquires the reference work time value corresponding to the class of the frame sequence 20, and calculates the allowable work time value based on the acquired reference work time value. The allowable work time value is calculated, for example, by multiplying the reference work time value by a predetermined constant greater than 1, or by adding a predetermined value to the reference work time value.

[0099] The calculation unit 2040 determines whether the length of the task time represented by the frame sequence 20 is equal to or less than the allowable task time calculated for the class of the frame sequence 20. Here, the length of the task time represented by the frame sequence 20 can be expressed, for example, as the difference between the generation time of the last video frame 12 in the frame sequence 20 and the generation time frame of the first video frame 12 in the frame sequence 20. If the length of the task time is equal to or less than the allowable value, the calculation unit 2040 determines that the length of the task time is appropriate. On the other hand, if the length of the task time exceeds the allowable value, the calculation unit 2040 determines that the length of the task time is inappropriate.

[0100] In the above example, the allowable task time represents the upper limit of the allowable task time. However, the calculation unit 2040 may use the lower limit of the allowable task time in addition to the upper limit of the allowable task time. In this case, the task time is determined to be appropriate if the task time represented by the frame sequence 20 is equal to or greater than the lower limit of the allowable task time and equal to or less than the upper limit of the allowable task time. The lower limit of the allowable task time is calculated, for example, by multiplying the reference task time value by a predetermined constant less than 1 or by subtracting a predetermined value from the reference task time value.

[0101] <Method for calculating repetitive action evaluation value> A single task may include a repetitive action that is repeated multiple times. Therefore, for example, for each frame sequence 20 included in the video data 10, the calculation unit 2040 determines whether the number of repetitive actions in the task represented by the frame sequence 20 is appropriate, and calculates a repetitive action evaluation value based on the result of the determination.

[0102] To determine whether the number of repetitions is appropriate, the calculation unit 2040 calculates an allowable number of repetitions (hereinafter, the allowable number of repetitions). For example, an appropriate number of repetitions (hereinafter, the reference number of repetitions) is determined in advance for each class. The calculation unit 2040 acquires the reference number of repetitions corresponding to the class of the frame sequence 20, and calculates the allowable number of repetitions based on the acquired reference number of repetitions. The allowable number of repetitions is calculated, for example, by multiplying the reference number of repetitions by a predetermined constant smaller than 1, or by subtracting a predetermined number from the reference number of repetitions.

[0103] Here, there may be tasks that do not include repetitive movements. Therefore, for a class of tasks that do not include repetitive movements, for example, a reference value for the number of repetitions is set to zero. Then, for example, a frame sequence 20 that belongs to a class in which the reference value for the number of repetitions is set to zero is excluded from the determination of whether the number of repetitive movements is appropriate.

[0104] The calculation unit 2040 determines whether the number of repetitive movements in the task represented by the frame sequence 20 is equal to or greater than the allowable number of repetitions calculated for the class of the frame sequence 20. If the number of repetitive movements is equal to or greater than the allowable value, the calculation unit 2040 determines that the number of repetitive movements is appropriate. On the other hand, if the number of repetitive movements is less than the allowable value, the calculation unit 2040 determines that the number of repetitive movements is inappropriate.

[0105] In the above example, the allowable number of repetitions represents the lower limit of the allowable number of repetitions. However, the calculation unit 2040 may use the upper limit of the allowable number of repetitions in addition to the lower limit of the allowable number of repetitions. In this case, if the number of repetitive movements in the task represented by the frame sequence 20 is equal to or greater than the lower limit of the allowable number of repetitions and equal to or less than the upper limit of the allowable number of repetitions, the number of repetitive movements is determined to be appropriate. Note that the upper limit of the allowable number of repetitions is calculated, for example, by multiplying the reference value of the number of repetitions by a predetermined constant greater than 1 or by adding a predetermined value to the reference value of the number of repetitions.

[0106] There are various methods for calculating the number of repetitive movements for the frame sequence 20. For example, the calculation unit 2040 calculates the number of repetitive movements by performing a process of detecting periodicity on the second real time series data.

[0107] <Output of Results> The task evaluation device 2000 outputs execution results. Hereinafter, information output from the task evaluation device 2000 will be referred to as output information. For example, the output information indicates an individual task evaluation value calculated for each frame sequence 20. Here, if a statistical value of the first evaluation value and the second evaluation value is calculated as the individual task evaluation value, the output information may indicate only the individual task evaluation value, or may indicate the first evaluation value and the second evaluation value in addition to the individual task evaluation value.

[0108] When the overall evaluation value is calculated using a sequence evaluation value and a time evaluation value, the output information indicates the overall evaluation value. In this case, the output information may further indicate each evaluation value used in calculating the overall evaluation value, such as an individual task evaluation value, a sequence evaluation value, a time evaluation value, and a repetitive action evaluation value.

[0109] The output information may be screen data (hereinafter, referred to as the analysis result screen) showing the analysis results of the video data 10. The analysis result screen shows the analysis results of the video data 10 in the form of text, images, graphs, or the like. More specifically, for example, the analysis result screen shows in text points that need to be improved in the work performed by the worker. As another example, the analysis result screen may have both the video data 10 and video data of ideal work embedded in a replayable form. As another example, the analysis result screen may show in graphs both a permutation representing the order of the work represented by the video data 10 and a permutation representing the correct work.

[0110] The output information may be output in any manner. For example, the work evaluation device 2000 may store the output information in any storage device. Alternatively, for example, the work evaluation device 2000 may transmit the output information to any device. Alternatively, for example, the work evaluation device 2000 may display the output information on any display device.

[0111] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0112] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0113] In the present disclosure, a program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0114] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) An task evaluation device comprising: an acquisition means for acquiring video data including a plurality of frame sequences each consisting of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; and a calculation means for calculating, for each of the frame sequences, a first evaluation value based on a change over time in the size of an image region of a tool detected from the frame sequence, a second evaluation value based on a change over time in the degree of bending of a finger detected from the frame sequence, or both the first evaluation value and the second evaluation value, thereby calculating an individual task evaluation value representing a level of evaluation of the task captured in the frame sequence, wherein the class represents a type of task. (Supplementary Note 2) The task evaluation device according to Supplementary Note 1, wherein the calculation means: detects an image region representing the tool from each video frame constituting the frame sequence; acquires time-series data representing a reference for the change over time in the size of the image region of the tool, the reference for the class of the frame sequence; and calculates the first evaluation value based on a similarity between the time-series data of the size of each of the detected image regions and the acquired reference time-series data. (Supplementary Note 3) The task evaluation device according to Supplementary Note 2, wherein the calculation means detects, as the tool, an object of a specific type that is associated with the class of the frame sequence. (Supplementary Note 4) The task evaluation device according to Supplementary Note 1, wherein the calculation means detects a specific finger from each video frame constituting the frame sequence, acquires time-series data representing a reference for time-varying changes in the degree of bending of the specific finger that is associated with the class of the frame sequence, and calculates the second evaluation value based on a similarity between the time-series data of the degree of bending of each of the detected specific fingers and the acquired reference time-series data. (Supplementary Note 5) The task evaluation device according to Supplementary Note 4, wherein the calculation means detects joint points of a hand of a person captured in the video frames, and calculates, as a value representing the degree of bending of the specific finger, a ratio of the distance between the joint point of the tip of the specific finger and the joint point of a wrist to the distance between the joint point of the tip of the specific finger and the joint point of a wrist when the specific finger is not bent.(Supplementary Note 6) The task evaluation device according to any one of Supplements 1 to 5, wherein the calculation means calculates a statistical value of the first evaluation value and the second evaluation value as the individual task evaluation value. (Supplementary Note 7) The task evaluation device according to any one of Supplements 1 to 5, wherein the calculation means calculates the individual task evaluation value for each of the plurality of frame sequences included in the video data, and calculates the statistical value of the individual task evaluation values ​​calculated for each of the frame sequences as the evaluation value of the plurality of tasks captured in the video data. (Supplementary Note 8) The task evaluation device according to any one of Supplements 1 to 5, wherein the calculation means calculates the evaluation values ​​of the plurality of tasks captured in the video data by calculating statistical values ​​of the individual task evaluation value calculated for each of the frame sequences and one or more of an evaluation value representing the appropriateness of the order of the plurality of tasks captured in the video data, an evaluation value representing the appropriateness of task times for the plurality of tasks captured in the video data, and an evaluation value representing the appropriateness of repetitive movements for the plurality of tasks captured in the video data. (Supplementary Note 9) A computer-executed task evaluation method, comprising: an acquisition step of acquiring video data including a plurality of frame sequences each consisting of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; and a calculation step of calculating, for each of the frame sequences, a first evaluation value based on a change over time in the size of an image area of ​​a tool detected from the frame sequence, a second evaluation value based on a change over time in the degree of bending of a finger detected from the frame sequence, or both the first evaluation value and the second evaluation value, thereby calculating an individual task evaluation value representing a level of evaluation of the task captured in the frame sequence, wherein the class represents a type of task.(Supplementary Note 10) A program that causes a computer to execute the following steps: an acquisition step of acquiring video data including a plurality of frame sequences each consisting of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; and a calculation step of calculating, for each of the frame sequences, a first evaluation value based on a change over time in the size of an image area of ​​a tool detected from the frame sequence, a second evaluation value based on a change over time in the degree of bending of a finger detected from the frame sequence, or both the first evaluation value and the second evaluation value, thereby calculating an individual task evaluation value that represents the level of evaluation of the task captured in the frame sequence, wherein the class represents a type of task.

[0115] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 8 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 9 and 10 in the same dependency relationship as Supplementary Notes 2 to 8. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0116] This application claims priority based on Japanese Patent Application No. 2023-193900, filed November 14, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0117] REFERENCE SIGNS LIST 10 Video data 12 Video frame 20 Frame sequence 200 Table 202 Frame identification information 204 Class identification information 300 Table 302 First frame identification information 304 Last frame identification information 306 Class identification information 400 Class definition information 402 Class identification information 404 Work type 406 Tool identification information 408 First reference time series data 410 Second reference time series data 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Work evaluation device 2020 Acquisition unit 2040 Calculation unit

Claims

1. A task evaluation device comprising: an acquisition means for acquiring video data including a plurality of frame sequences each consisting of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; and a calculation means for calculating, for each of the frame sequences, a first evaluation value based on the change over time in size of an image area of ​​a tool detected from the frame sequence, a second evaluation value based on the change over time in the degree of bending of a finger detected from the frame sequence, or both the first evaluation value and the second evaluation value, thereby calculating an individual task evaluation value representing the level of evaluation of the task imaged in the frame sequence, wherein the class represents a type of task.

2. The work evaluation device described in claim 1, wherein the calculation means detects an image area representing the tool from each video frame constituting the frame sequence, acquires time series data representing a standard of time change in size of the image area of ​​the tool that is associated with the class of the frame sequence, and calculates the first evaluation value based on the similarity between the time series data of the size of each of the detected image areas and the acquired standard time series data.

3. The task evaluation device according to claim 2, wherein the calculation means detects, as the tool, an object of a specific type associated with the class of the frame sequence.

4. The work evaluation device described in claim 1, wherein the calculation means detects a specific finger from each video frame constituting the frame sequence, acquires time series data representing a standard of time change in the degree of bending of the specific finger that is associated with the class of the frame sequence, and calculates the second evaluation value based on the similarity between the time series data of the degree of bending of each of the detected specific fingers and the acquired standard time series data.

5. The work evaluation device described in claim 4, wherein the calculation means detects joint points of the hand of the person captured in the video frame, and calculates a ratio of the distance between the joint point of the tip of the specific finger and the joint point of the wrist to the distance between the joint point of the tip of the specific finger and the joint point of the wrist in an unbent state, as a value representing the degree of bending of the specific finger.

6. The task evaluation device according to claim 1, wherein the calculation means calculates a statistical value of the first evaluation value and the second evaluation value as the individual task evaluation value.

7. A task evaluation device as described in any one of claims 1 to 5, wherein the calculation means calculates the individual task evaluation value for each of the plurality of frame sequences included in the video data, and calculates a statistical value of the individual task evaluation values ​​calculated for each of the frame sequences as an evaluation value for the plurality of tasks captured in the video data.

8. A task evaluation device as described in any one of claims 1 to 5, wherein the calculation means calculates evaluation values ​​for the multiple tasks captured in the video data by calculating statistical values ​​for one or more of an evaluation value representing the appropriateness of the order of the multiple tasks captured in the video data, an evaluation value representing the appropriateness of the task time for the multiple tasks captured in the video data, and an evaluation value representing the appropriateness of repetitive movements for the multiple tasks captured in the video data, and the individual task evaluation values ​​calculated for each of the frame sequences.

9. A computer-executed task evaluation method comprising: an acquisition step of acquiring video data including a plurality of frame sequences each consisting of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; and a calculation step of calculating, for each of the frame sequences, an individual task evaluation value representing the level of evaluation of the task captured in the frame sequences by calculating a first evaluation value based on the change over time in size of the image area of ​​a tool detected from the frame sequence, a second evaluation value based on the change over time in the degree of bending of a finger detected from the frame sequence, or both the first evaluation value and the second evaluation value, wherein the class represents a type of task.

10. A program that causes a computer to execute the following steps: an acquisition step of acquiring video data including a plurality of frame sequences each consisting of a plurality of consecutive video frames belonging to the same class, wherein adjacent frame sequences belong to different classes; and a calculation step of calculating, for each of the frame sequences, a first evaluation value based on the change over time in the size of the image area of ​​a tool detected from the frame sequence, a second evaluation value based on the change over time in the degree of bending of a finger detected from the frame sequence, or both the first evaluation value and the second evaluation value, thereby calculating an individual task evaluation value that represents the level of evaluation of the task imaged in the frame sequence, wherein the class represents a type of task.