Work recognition device, work recognition method, and program

Abstract

This work recognition device comprises: a training unit that generates a trained model by conducting machine learning using teaching data; and a recognition unit that recognizes a work type using the trained model. The training unit generates a first inter-frame difference image on the basis of a frame included in a first unit section of a moving image for training, and uses, as teaching data, data indicating the first inter-frame difference image and a label indicating the work type assigned to the first unit section. The recognition unit generates a second inter-frame difference image on the basis of the frame included in a second unit section of the moving image to be recognized, and recognizes the work type of the worker appearing in the second unit section on the basis of the output of the trained model when the second inter-frame difference image is inputted. Thus, a work recognition device having a high degree of freedom in the installation place of a camera and high recognition accuracy is provided.

Description

Task recognition device, task recognition method, and program

[0001] The present disclosure relates to an activity recognition device, an activity recognition method, and a program.

[0002] Conventionally, a technique has been developed for recognizing the type of work performed by a worker from a video obtained by filming the worker.

[0003] China Patent Publication No. 109409209 (Patent Document 1) discloses a method including the steps of acquiring human body video information, preprocessing the human body video information to acquire skeletal node information indicating the positions of joint points, and inputting the skeletal node information into an action recognition model to identify actions.

[0004] Japanese Patent Laid-Open Publication No. 2021-67981 (Patent Document 2) discloses a device that generates a model that takes into account the association between work-related objects and elemental work by using image information including an imaged area of ​​work-related objects related to an elemental work identified by elemental work information as training data. Based on the model, the device identifies which elemental work the target image is related to the worker performing.

[0005] Chinese Patent Application Publication No. 109409209 Patent Publication No. 2021-67981

[0006] The method described in Patent Literature 1 identifies human behavior based on skeleton node information acquired by preprocessing human body video information. Therefore, it is necessary to generate human body video information that shows the human body to an extent that the skeleton can be recognized. Therefore, there are limitations on where a camera that generates human body video information can be installed.

[0007] On the other hand, the device described in Patent Document 2 directly inputs a target image into a model without extracting skeletal node information. Therefore, there are no restrictions on the installation location of the camera that generates the target image. However, the target image captures not only the worker but also the background surrounding the worker. Therefore, based on the characteristics of the background captured in the target image, the model may mistakenly recognize a target image that does not capture a worker as an image of a worker performing a specific elemental task. Alternatively, based on the characteristics of the worker and the background, the model may mistakenly recognize a target image that captures a worker waiting to perform a task as an image of a worker performing a specific elemental task.

[0008] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide an activity recognition device, activity recognition method, and program that have a high degree of freedom in the location of the camera that acquires images and have high recognition accuracy.

[0009] According to an example of the present disclosure, an activity recognition device that recognizes an activity type of a worker includes a learning unit that generates a trained model by performing machine learning using training data, and a recognition unit that recognizes the activity type using the trained model. The learning unit generates a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming the worker's work site. The learning unit uses data indicating the first inter-frame difference image and labels indicating the activity type assigned to the frames included in the first unit section as training data. The recognition unit generates a second inter-frame difference image based on frames included in a second unit section of a recognition-target video obtained by filming the work site. The recognition unit recognizes the activity type of the worker appearing in the second unit section of the recognition-target video based on the output of the trained model when the second inter-frame difference image is input.

[0010] According to this disclosure, the trained model receives as input inter-frame difference images in unit sections. Generally, inter-frame difference images in unit sections where a worker is performing a task differ significantly from inter-frame difference images in unit sections where the worker is absent or where the worker is waiting to perform a task. Therefore, the task recognition device can accurately recognize the task type of the worker appearing in the second unit section by inputting a second inter-frame difference image in a second unit section of the recognition target video into the trained model. Furthermore, the trained model does not require input of skeleton node information extracted from the image, as in the technology disclosed in Patent Literature 1. Therefore, there are no restrictions on the installation location of the camera, as in the technology disclosed in Patent Literature 1. In other words, there is a high degree of freedom in the installation location of the camera for acquiring the training images and the recognition target images.

[0011] In the above disclosure, the activity recognition device further includes a person detection unit that detects a person from each frame of the recognition target video. The recognition unit recognizes the activity type only for frames in which a person is detected. This disclosure reduces the load on the recognition unit.

[0012] For example, when the first unit interval and the second unit interval each include two consecutive frames, the first inter-frame difference image may be a difference image between the two frames included in the first unit interval, and the second inter-frame difference image may be a difference image between the two frames included in the second unit interval.

[0013] Alternatively, the first inter-frame difference image may be a difference image between one of two frames included in the first unit section and an interpolated frame inserted between the two frames included in the first unit section, and the second inter-frame difference image may be a difference image between one of two frames included in the second unit section and an interpolated frame inserted between the two frames included in the second unit section.

[0014] Alternatively, when each of the first unit interval and the second unit interval includes three or more consecutive frames, the first inter-frame difference image may be a difference image between the first frame and the last frame of the three or more frames included in the first unit interval, and the second inter-frame difference image may be a difference image between the first frame and the last frame of the three or more frames included in the second unit interval.

[0015] Alternatively, when each of the first unit interval and the second unit interval includes four or more consecutive frames, the first inter-frame difference image may be a difference image between a frame obtained by weighting the first half of the four or more frames included in the first unit interval and a frame obtained by weighting the second half of the four or more frames included in the first unit interval, and the second inter-frame difference image may be a difference image between a frame obtained by weighting the first half of the four or more frames included in the second unit interval and a frame obtained by weighting the second half of the four or more frames included in the second unit interval.

[0016] In the above disclosure, the learning unit generates a trained model using few-shot learning. According to this disclosure, a trained model can be generated using a small amount of training data.

[0017] According to an example of the present disclosure, an activity recognition method for recognizing an activity type of a worker includes: generating a trained model by one or more processors performing machine learning using training data; and recognizing the activity type using the trained model by the one or more processors. Generating the trained model includes generating a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming the worker's work site, and using data indicating the first inter-frame difference image and labels indicating the activity type assigned to the frames included in the first unit section as training data. Recognizing includes generating a second inter-frame difference image based on frames included in a second unit section of a recognition-target video obtained by filming the work site, and recognizing the activity type of the worker appearing in the second unit section of the recognition-target video based on output of the trained model when the second inter-frame difference image is input.

[0018] According to yet another example of the present disclosure, a program causes a computer to execute the above information processing method.

[0019] These disclosures also provide a high degree of freedom in the location of installation of the camera that captures images, and a high degree of accuracy in recognizing tasks.

[0020] According to the present disclosure, there is a high degree of freedom in the location of installation of a camera that captures images, and the accuracy of task recognition is high.

[0021] FIG. 1 is a diagram schematically illustrating an example of a system to which an activity recognition device according to an embodiment is applied. FIG. 2 is a flowchart illustrating an example of the flow of a trained model generation process. FIG. 3 is a diagram illustrating a method for generating training data. FIG. 4 is a diagram illustrating another example of a method for generating training data. FIG. 5 is a diagram illustrating a first example of an inter-frame difference image generation. FIG. 6 is a diagram illustrating a second example of an inter-frame difference image generation. FIG. 7 is a diagram illustrating a third example of an inter-frame difference image generation. FIG. 8 is a diagram illustrating a fourth example of an inter-frame difference image generation. FIG. 9 is a flowchart illustrating an example of the flow of an activity type recognition process. FIG. 10 is a flowchart illustrating another example of the flow of an activity type recognition process. FIG. 11 is a diagram illustrating an example of an activity type recognition error.

[0022] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals and the description thereof will not be repeated.

[0023] §1 Reference Form Before describing the embodiments of the present disclosure, a reference form and its problems will be described. Similar to, for example, Patent Document 2, an activity recognition device according to the reference form uses a trained model that receives input of video frames obtained by filming the workplace of a worker and outputs the probability that the frame will be classified into each of a plurality of activity types.

[0024] The trained model used by the task recognition device according to the reference embodiment receives video frames directly as input. Therefore, the task recognition device according to the reference embodiment has the advantage of not requiring the preprocessing required to acquire skeleton node information, as described in Patent Document 1. However, task type recognition accuracy is likely to be low. In particular, video footage of a work site often includes various objects other than the worker. This influence can easily result in low recognition accuracy.

[0025] 11 is a diagram showing an example of a recognition error of a task type. In the example shown in FIG. 11, the trained model is generated using training data that shows an image 60 of a worker 13 performing a task of task type "A" and a classification label that represents task type "A." Task type "A" represents a task performed by the worker 13 on a board 70.

[0026] Image 61 does not include worker 13. However, if image 61 is similar to image 60 except for the presence or absence of worker 13, the trained model may recognize the type of work shown in image 61 as work type "A."

[0027] Image 62 shows a state in which worker 13 is waiting for board 70. However, the state of worker 13 in image 62 is similar to the state of worker 13 in image 60. Therefore, the trained model can recognize the type of work shown in image 62 as work type "A."

[0028] As described above, the trained model according to the reference embodiment, which receives as input frames of a video captured at a work site, has low recognition accuracy. In particular, the trained model according to the reference embodiment may erroneously recognize the task type in frames in which no worker is present or in which the worker is waiting to perform the task.

[0029] In contrast, the task recognition device according to this embodiment can improve the recognition accuracy for frames in which no worker is present or in which the worker is waiting to perform a task, as will be described later.

[0030] §2 Application Example Fig. 1 is a diagram schematically illustrating an example of a system to which an activity recognition device according to an embodiment is applied. The system 1 shown in Fig. 1 includes a production site 10, an equipment management server 20, an activity recognition device 30, and a visualization server 40. Each of the equipment management server 20, the activity recognition device 30, and the visualization server 40 may be configured with one or more computers, and may include a virtual machine or a container built in a cloud environment, or a configuration consisting of at least a part of these.

[0031] One or more pieces of equipment 11 and one or more network cameras 12 are installed at the production site 10. The one or more pieces of equipment 11 include, for example, manufacturing equipment, sensors, conveyance equipment, etc. The one or more pieces of equipment 11 output equipment signals indicating their operating states to the equipment management server 20 at predetermined control cycles.

[0032] One or more network cameras 12 capture images of workers 13 performing work at the production site 10. The one or more network cameras 12 output video data obtained by capturing the images (hereinafter simply referred to as "video") to the task recognition device 30. Each frame of the video may be a color image or a monochrome image.

[0033] The equipment management server 20 receives equipment signals from one or more pieces of equipment 11 and manages the values ​​of one or more variables that indicate the state of the one or more pieces of equipment 11. The equipment management server 20 includes an equipment database 21 that stores the values ​​of the one or more variables for each control period.

[0034] The task recognition device 30 recognizes tasks performed by workers 13 at the production site 10. The task recognition device 30 is configured by a computer with a general-purpose architecture. The task recognition device 30 includes, as its main components, a processor 31, a memory 32, and a user interface 38. The processor 31 includes, for example, a central processing unit (CPU) or a micro processing unit (MPU). The memory 32 includes, for example, a volatile storage device such as a dynamic random access memory (DRAM). The user interface 38 includes a display and an input device. The input device may include, for example, a keyboard, a mouse, or a touchpad.

[0035] The task recognition device 30 further includes a recording service 35, an task recognition application 36, and an annotation tool 37 as programs executed by the processor 31. These programs are stored in a storage device such as a hard disk or a storage medium. Storage media include volatile storage media, nonvolatile storage media, general-purpose semiconductor storage devices such as Compact Flash (CF) or Secure Digital (SD), magnetic storage media such as flexible disks, and optical storage media such as compact disk read-only memories (CD-ROMs). Some or all of the functions provided by these programs may be realized by dedicated hardware circuits (e.g., application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs)). The task recognition device 30 further includes a task recognition database 34.

[0036] The recording service 35 outputs a recording instruction to the network camera 12 and causes the processor 31 to execute an operation of collecting videos obtained by shooting in accordance with the recording instruction. The videos collected by executing the recording service 35 include learning videos for generating teacher data and recognition target videos that are targets for task type recognition.

[0037] The annotation tool 37 causes the processor 31 to perform an operation of assigning a classification label selected from a plurality of classification labels representing task types to each frame of the learning video in response to an input to the user interface 38. The annotation tool 37 causes the processor 31 to perform an operation of generating annotation data indicating the classification labels assigned to each frame of the learning video.

[0038] The task recognition application 36 causes the processor 31 to execute a process for generating a trained model, a process for recognizing task types using the trained model, and a process for storing the task type recognition results in the task recognition database 34. When the processor 31 executes the task recognition application 36, a learning unit 311, a recognition unit 312, and a human detection unit 313 are realized.

[0039] In addition to the recognition results of the task type, the task recognition database 34 may store equipment data indicating some or all of the values ​​of one or more variables for each control period stored in the equipment database 21 .

[0040] The learning unit 311 generates a trained model by performing machine learning using training data. The training data is generated based on annotation data. Specifically, the learning unit 311 generates a first inter-frame difference image based on frames included in a first unit section of the training video. The learning unit 311 uses data indicating the first inter-frame difference image and classification labels assigned to the frames included in the first unit section as training data.

[0041] In general machine learning, a trained model is generated using a large amount of training data. In contrast, in few-shot learning, a trained model is generated using a small amount of training data. In general, the recognition accuracy of a trained model generated using few-shot learning is lower than the recognition accuracy of a trained model generated using general machine learning. However, as will be described later, the task recognition accuracy of the task recognition device 30 according to this embodiment is high. Therefore, the learning unit 311 may generate a trained model using few-shot learning.

[0042] The recognition unit 312 uses the trained model to recognize the type of work shown in each frame of the recognition target image. Specifically, the recognition unit 312 generates a second inter-frame difference image based on frames included in a second unit segment of the recognition target video obtained by filming the work site of the worker 13. The recognition unit 312 recognizes the type of work of the worker shown in the second unit segment based on the output of the trained model when the second inter-frame difference image is input. In the following description, when the first unit segment and the second unit segment are not distinguished from each other, each of the first unit segment and the second unit segment is simply referred to as a "unit segment." When the first inter-frame difference image and the second inter-frame difference image are not distinguished from each other, each of the first inter-frame difference image and the second inter-frame difference image is simply referred to as an "inter-frame difference image."

[0043] The human detection unit 313 detects a human from each frame of the recognition target video. The detection result by the human detection unit 313 can be used to select frames to be recognized as work types.

[0044] The visualization server 40 provides a visualization service 41 that visualizes the recognition results obtained by the task recognition device 30. For example, in response to a request from a user terminal (not shown), the visualization service 41 accesses the task recognition database 34 to obtain data to be visualized (visualization data). The visualization service 41 generates a screen showing the task type recognition results based on the visualization data and outputs the generated screen to the user terminal. This allows the user to check the task type recognition results using the user terminal.

[0045] The processing of the processor operating as a learning unit will be described with reference to Figures 2 to 8. Figure 2 is a flowchart showing an example of the flow of processing for generating a trained model. Figure 3 is a diagram illustrating an example of a method for generating training data. Figure 4 is a diagram illustrating another example of a method for generating training data. Figure 5 is a diagram illustrating a first example of generating an inter-frame difference image. Figure 6 is a diagram illustrating a second example of generating an inter-frame difference image. Figure 7 is a diagram illustrating a third example of generating an inter-frame difference image. Figure 8 is a diagram illustrating a fourth example of generating an inter-frame difference image.

[0046] First, in step S1, the processor 31 operating as the learning unit 311 acquires annotation data. As shown in Figures 3 and 4, the annotation data 80 indicates classification labels assigned to each frame of the learning video.

[0047] In the next step S2, the processor 31 generates a first inter-frame difference image based on frames included in a first unit interval of the training video. The time length of the first unit interval is determined in advance according to the frame rate of the training video. For example, the first unit interval may have a time length including two consecutive frames. Alternatively, the first unit interval may have a time length including three or more consecutive frames.

[0048] The method of generating training data shown in Fig. 3 and the examples of generating inter-frame difference images shown in Figs. 5 and 6 correspond to a unit section having a time length including two consecutive frames. The method of generating training data shown in Fig. 4 and the examples of generating inter-frame difference images shown in Figs. 7 and 8 correspond to a unit section having a time length including three or more consecutive frames.

[0049] When a unit section having a time length including two consecutive frames is set, the processor 31 may generate a difference image between the two frames included in the unit section as an inter-frame difference image. For example, as shown in FIG. 5 , the processor 31 generates a difference image 52 between the (k-1)th frame 50 and the kth frame 51 of the video as an inter-frame difference image. Then, as shown in FIG. 3 , for each first unit section of the training video, the processor 31 uses, as training data 85, data indicating the first inter-frame difference image and classification labels assigned to the frames included in the first unit section. In the example shown in FIG. 3 , the processor 31 generates training data 85 indicating a difference image "1, 2" between the first and second frames and a classification label "A." Furthermore, the processor 31 generates training data 85 indicating a difference image "2, 3" between the second and third frames and a classification label "A." Furthermore, the processor 31 generates training data 85 indicating a difference image "N-1, N" between the (N-1)th frame and the Nth frame and a classification label "C."

[0050] Alternatively, as shown in FIG. 6 , the processor 31 may insert an interpolated frame 53 between the (k−1)th frame 50 and the kth frame 51 of the video. The processor 31 generates the interpolated frame 53 using a known technique. The processor 31 may then generate a difference image 54 between the frame 50 and the interpolated frame 53 as an inter-frame difference image. Alternatively, the processor 31 may generate a difference image between the interpolated frame 53 and the frame 51 as an inter-frame difference image instead of or in addition to the difference image 54. The processor 31 then uses, for each first unit section of the training video, data indicating the generated first inter-frame difference image and the classification labels assigned to the frames included in that first unit section as training data 85.

[0051] When a unit interval having a time length including three or more consecutive frames is set, the processor 31 may generate, as an inter-frame difference image, a difference image between the first frame and the last frame in the unit interval. For example, as shown in Fig. 7, the processor 31 generates, as an inter-frame difference image, a difference image 56 between the (k-2)th frame 55 and the kth frame 51 of the video. The (k-1)th frame 50 is thinned out and is not used to generate the inter-frame difference image.

[0052] Alternatively, processor 31 may divide the four or more frames included in the unit section into a first half frame group and a second half frame group. Then, processor 31 may generate, as an inter-frame difference image, a difference image between a frame obtained by weighting the first half frame group and a frame obtained by weighting the second half frame group. For example, as shown in FIG. 8 , processor 31 generates, as an inter-frame difference image, a difference image 59 between frame 57a obtained by weighting the k-5th to k-3th frame group 57 included in the unit section and frame 58a obtained by weighting the k-2nd to kth frame group 58 included in the unit section.

[0053] The processor 31 uses data indicating a first inter-frame difference image generated based on three or more frames included in each first unit section of the learning video and classification labels assigned to the frames included in the first unit section as training data 85. In the example shown in FIG. 4 , the processor 31 generates training data 85 indicating a difference image "1, p" generated from the 1st to pth frames and a classification label "A." Furthermore, the processor 31 generates training data 85 indicating a difference image "N-p+1, N" generated from the N-p+1st to Nth frames and a classification label "C."

[0054] In addition, if different classification labels are assigned to multiple frames included in a first unit section, the processor 31 may omit generating the training data 85 for that first unit section.

[0055] In the next step S3, the processor 31 generates a trained model by performing machine learning using the training data 85. The trained model receives an input of a difference image between two consecutive frames and outputs the probability that the difference image is classified into each of a plurality of classification labels. The trained model may be generated using few-shot learning. This reduces the amount of training data 85 required to generate the trained model.

[0056] In the next step S4, the processor 31 outputs the generated trained model. Specifically, the processor 31 stores the trained model in the memory 32 or a storage device (not shown). After step S4, the processing of the processor 31 operating as the training unit 311 ends.

[0057] The work type recognition process will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the flow of the work type recognition process.

[0058] First, in step S11, the processor 31 operating as the recognition unit 312 reads out a trained model. In step S12, the processor 31 acquires a video to be recognized. The video to be recognized is output from the network camera 12. The order of steps S11 and S12 is not particularly limited.

[0059] In the next step S13, the processor 31 generates a second inter-frame difference image based on frames included in a second unit interval of the recognition target video. The method for generating the second inter-frame difference image is the same as the method for generating the first inter-frame difference image. Note that if the frame rates of the recognition target video and the learning video are the same, the first unit interval and the second unit interval have the same time length, and the first inter-frame difference image and the second inter-frame difference image are generated by the same method. For example, if the first inter-frame difference image is generated according to the first generation example shown in FIG. 5, the second inter-frame difference image is also generated according to the first generation example.

[0060] In the next step S14, the processor 31 recognizes the task type of the worker 13 appearing in the second unit section corresponding to the second frame difference image, based on the output of the trained model when the second frame difference image is input. As described above, the trained model receives the frame difference image and outputs the probability that the frame difference image will be classified into each of multiple classification labels. Therefore, the processor 31 recognizes the task type represented by the classification label corresponding to the highest probability output from the trained model as the task type of the worker 13 appearing in the second unit section.

[0061] Steps S13 and S14 are performed for all second unit segments included in the authentication target video. When steps S13 and S14 have been performed for all second unit segments, the process proceeds to step S15.

[0062] In step S15, the processor 31 stores the recognition result for each second unit section in the task recognition database 34. After step S15, the process of the processor 31 operating as the recognition unit 312 ends.

[0063] During a period when the worker 13 is performing a task, the worker 13's hands and the object being worked on typically move. Therefore, as shown in FIG. 5 , the difference image 52 exhibits high brightness in moving portions. In contrast, no movement is observed in the video during unit sections in which the worker 13 is absent. Furthermore, no movement is typically observed in the video during unit sections in which the worker 13 is waiting. Therefore, the frame-to-frame difference image during a unit section in which the worker 13 is performing a task differs significantly from the frame-to-frame difference image during a unit section in which the worker 13 is absent or waiting. Therefore, by using a trained model generated by performing machine learning using training data indicating the frame-to-frame difference image and the classification label, the task recognition device 30 can accurately recognize the task type depicted in the unit section of the video to be recognized. In other words, erroneous recognition of frames included in unit sections in which the worker 13 is absent or waiting as frames showing the worker 13 performing a task is suppressed.

[0064] Furthermore, the trained model according to this embodiment receives input of an inter-frame difference image, and does not require input of skeleton node information extracted from the image as in the technology disclosed in Patent Document 1. Therefore, unlike the technology disclosed in Patent Document 1, there are no restrictions on where the camera can be installed, and there is a high degree of freedom in where the network camera 12 can be installed.

[0065] Fig. 10 is a flowchart showing another example of the flow of the task type recognition process. The flow shown in Fig. 10 differs from the flow shown in Fig. 9 in that step S20 is included between step S12 and step S13.

[0066] In step S20, the processor 31 operating as the human detection unit 313 detects a human from each frame of the video to be recognized. The processor 31 may use known object recognition technology to determine whether a human is present in each frame. The processor 31 excludes frames in which a human is not detected from the processing targets of step S13 and thereafter. In other words, the processor 31 executes the processing of step S13 and thereafter only for frames in which a human is detected. As a result, the processor 31 operating as the recognition unit 312 recognizes the task type only for frames in which a human is detected. As a result, the load on the processor 31 is reduced.

[0067] §3 Supplementary Note As described above, the present embodiment includes the following disclosure.

[0068] (Configuration 1) A task recognition device (30) that recognizes a task type of a worker (13), comprising: a learning unit (31, 311) that generates a trained model by performing machine learning using teacher data (85); and a recognition unit (31, 312) that recognizes the task type using the trained model, wherein the learning unit (31, 311) generates a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming a work site of the worker, and uses data indicating the first inter-frame difference image and labels that indicate the task type assigned to frames included in the first unit section as the teacher data (85), and the recognition unit (31, 312) generates a second inter-frame difference image based on frames included in a second unit section of a recognition target video obtained by filming the work site, and recognizes the task type of the worker appearing in the second unit section based on output of the trained model when the second inter-frame difference image is input.

[0069] (Configuration 2) The task recognition device (30) according to Configuration 1, further comprising a person detection unit (31, 313) that detects a person from each frame of the recognition target video, wherein the recognition unit (31, 312) recognizes the task type only for frames in which a person is detected.

[0070] (Configuration 3) The task recognition device (30) according to Configuration 1 or 2, wherein each of the first unit interval and the second unit interval includes two consecutive frames, the first inter-frame difference image is a difference image between the two frames included in the first unit interval, and the second inter-frame difference image is a difference image between the two frames included in the second unit interval.

[0071] (Configuration 4) The task recognition device (30) according to Configuration 1 or 2, wherein each of the first unit interval and the second unit interval includes two consecutive frames, the first inter-frame difference image is a difference image between one of the two frames included in the first unit interval and an interpolated frame inserted between the two frames included in the first unit interval, and the second inter-frame difference image is a difference image between one of the two frames included in the second unit interval and an interpolated frame inserted between the two frames included in the second unit interval.

[0072] (Configuration 5) The task recognition device (30) according to Configuration 1 or 2, wherein each of the first unit interval and the second unit interval includes three or more consecutive frames, the first inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the first unit interval, and the second inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the second unit interval.

[0073] (Configuration 6) The task recognition device (30) according to Configuration 1 or 2, wherein each of the first unit interval and the second unit interval includes four or more consecutive frames, the first inter-frame difference image is a difference image between a frame obtained by weighting an average of a first half of the four or more frames included in the first unit interval and a frame obtained by weighting an average of a second half of the four or more frames included in the first unit interval, and the second inter-frame difference image is a difference image between a frame obtained by weighting an average of a first half of the four or more frames included in the second unit interval and a frame obtained by weighting an average of a second half of the four or more frames included in the second unit interval.

[0074] (Configuration 7) The task recognition device (30) according to any one of Configurations 1 to 6, wherein the learning unit (31, 311) generates the trained model using few-shot learning.

[0075] (Configuration 8) A task recognition method for recognizing a task type of a worker (13), comprising: one or more processors (31) performing machine learning using teacher data (85) to generate a trained model (S1 to S4); and the one or more processors (31) recognizing the task type using the trained model (S11 to S15), wherein generating the trained model (S1 to S4) includes: generating a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming a work site of the worker (S2); and using data indicating the first inter-frame difference image and labels indicating the task type assigned to frames included in the first unit section as the teacher data (85) (S3), and wherein recognizing (S11 to S15) includes: generating a second inter-frame difference image based on frames included in a second unit section of a recognition target video obtained by filming the work site (S13); and recognizing (S14) the type of work of the worker appearing in the second unit section of the video to be recognized, based on the output of the trained model when the second inter-frame difference image is input.

[0076] (Configuration 9) A program (36) for causing a computer (31) to execute a task recognition method for recognizing a task type of a worker (13), the task recognition method comprising: generating a trained model by performing machine learning using training data (85) (S1 to S4); and recognizing the task type using the trained model (S11 to S15), wherein generating the trained model (S1 to S4) comprises: generating a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming a work site of the worker (S2); and using data indicating the first inter-frame difference image and labels indicating the task type assigned to frames included in the first unit section as the training data (S3), and wherein recognizing (S11 to S15) comprises: generating a second inter-frame difference image based on frames included in a second unit section of a recognition target video obtained by filming the work site (S13); and recognizing (S14) the type of work performed by the worker appearing in the second unit section of the video to be recognized, based on the output of the trained model when the second inter-frame difference image is input.

[0077] Although the embodiments of the present invention have been described, the embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims.

[0078] 1 System, 10 Production site, 11 Equipment, 12 Network camera, 13 Worker, 20 Equipment management server, 21 Equipment database, 30 Work recognition device, 31 Processor, 32 Memory, 34 Work recognition database, 35 Recording service, 36 Work recognition application, 37 Annotation tool, 38 User interface, 40 Visualization server, 41 Visualization service, 50, 51, 55, 57a, 58a Frame, 52, 54, 56, 59 Difference image, 53 Interpolated frame, 57, 58 Frame group, 60, 61, 62 Image, 70 Board, 80 Annotation data, 85 Teacher data, 311 Learning unit, 312 Recognition unit, 313 Human detection unit.

Claims

1. A task recognition device that recognizes a task type of a worker, comprising: a learning unit that generates a trained model by performing machine learning using training data; and a recognition unit that recognizes the task type using the trained model, wherein the learning unit: generates a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming the worker's work site; and uses data indicating the first inter-frame difference image and labels indicating the task type assigned to frames included in the first unit section as the training data; and the recognition unit: generates a second inter-frame difference image based on frames included in a second unit section of a recognition target video obtained by filming the work site; and recognizes the task type of the worker appearing in the second unit section based on the output of the trained model when the second inter-frame difference image is input.

2. The task recognition device according to claim 1, further comprising a person detection unit that detects a person from each frame of the video to be recognized, wherein the recognition unit recognizes the task type only for frames in which a person is detected.

3. A task recognition device as described in claim 1 or 2, wherein each of the first unit interval and the second unit interval includes two consecutive frames, the first inter-frame difference image is a difference image between the two frames included in the first unit interval, and the second inter-frame difference image is a difference image between the two frames included in the second unit interval.

4. A task recognition device as described in claim 1 or 2, wherein each of the first unit interval and the second unit interval includes two consecutive frames, the first inter-frame difference image is a difference image between one of the two frames included in the first unit interval and an interpolated frame inserted between the two frames included in the first unit interval, and the second inter-frame difference image is a difference image between one of the two frames included in the second unit interval and an interpolated frame inserted between the two frames included in the second unit interval.

5. The task recognition device according to claim 1 or 2, wherein each of the first unit interval and the second unit interval includes three or more consecutive frames, the first inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the first unit interval, and the second inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the second unit interval.

6. The task recognition device according to claim 1 or 2, wherein each of the first unit interval and the second unit interval includes four or more consecutive frames, the first inter-frame difference image is a difference image between a frame obtained by weighting the first half of the four or more frames included in the first unit interval and a frame obtained by weighting the second half of the four or more frames included in the first unit interval, and the second inter-frame difference image is a difference image between a frame obtained by weighting the first half of the four or more frames included in the second unit interval and a frame obtained by weighting the second half of the four or more frames included in the second unit interval.

7. The task recognition device according to any one of claims 1 to 6, wherein the learning unit generates the trained model using few-shot learning.

8. A task recognition method for recognizing a task type of a worker, comprising: one or more processors performing machine learning using training data to generate a trained model; and the one or more processors recognizing the task type using the trained model, wherein generating the trained model comprises: generating a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming the worker's work site; and using data indicating the first inter-frame difference image and labels indicating the task type assigned to frames included in the first unit section as the training data, and recognizing comprises: generating a second inter-frame difference image based on frames included in a second unit section of a recognition target video obtained by filming the work site; and recognizing the task type of the worker appearing in the second unit section of the recognition target video based on output of the trained model when the second inter-frame difference image is input.

9. A program causing a computer to execute an activity recognition method for recognizing an activity type of a worker, the activity recognition method comprising: generating a trained model by performing machine learning using training data; and recognizing the activity type using the trained model, wherein generating the trained model comprises: generating a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming the work site of the worker; and using data indicating the first inter-frame difference image and labels indicating the activity type assigned to frames included in the first unit section as the training data, and recognizing comprises: generating a second inter-frame difference image based on frames included in a second unit section of a recognition target video obtained by filming the work site; and recognizing the activity type of the worker appearing in the second unit section of the recognition target video based on output of the trained model when the second inter-frame difference image is input.

Images