Operation recognition device, operation recognition method and program

Abstract

To provide an operation recognition device with a high degree of freedom in a camera installation location and high recognition accuracy.SOLUTION: An operation recognition device comprises: a learning unit for generating a learned model by performing machine learning using teacher data; and a recognition unit for recognizing an operation type using the learned model. The learning unit generates a first inter-frame difference image based on frames included in a first unit section of a learning video, and uses data indicating the first inter-frame difference image and a label indicating an operation type assigned to the first unit section as teacher data. The recognition unit generates a second inter-frame difference image based on frames included in a second unit section of a video to be recognized, and, on the basis of output of the learned model when the second inter-frame difference image is input, recognizes an operation type for an operator captured in the second unit section.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an activity recognition device, an activity recognition method, and a program. [Background technology]

[0002] Conventionally, a technique has been developed for recognizing the type of work performed by a worker from a video obtained by filming the worker.

[0003] China Patent Application Publication No. 109409209 (Patent Document 1) discloses a method including the steps of acquiring human body video information, preprocessing the human body video information to acquire skeletal node information indicating the positions of joint points, and inputting the skeletal node information into an action recognition model to identify actions.

[0004] Japanese Patent Laid-Open Publication No. 2021-67981 (Patent Document 2) discloses a device that generates a model that takes into account the association between work-related objects and elemental work by using image information including an area capturing an image of a work-related object related to an elemental work identified by elemental work information as learning target data. Based on the model, the device identifies which elemental work the target image relates to the worker performing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Chinese Patent Application Publication No. 109409209 [Patent Document 2] Patent Publication No. 2021-67981 Summary of the Invention [Problem to be solved by the invention]

[0006] The method described in Patent Document 1 identifies human behavior based on skeleton node information acquired by preprocessing human body video information. Therefore, it is necessary to generate human body video information that shows the human body to an extent that the skeleton can be recognized. Therefore, there are limitations on where the camera that generates the human body video information can be installed.

[0007] On the other hand, the device described in Patent Document 2 directly inputs the target image into the model without extracting skeleton node information. Therefore, there are no restrictions on the installation location of the camera that generates the target image. However, the target image captures not only the worker but also the background surrounding the worker. Therefore, based on the characteristics of the background in the target image, the model may mistakenly recognize a target image that does not capture a worker as an image of a worker performing a specific elemental task. Alternatively, based on the characteristics of the worker and the background, the model may mistakenly recognize a target image that captures a worker waiting to perform a task as an image of a worker performing a specific elemental task.

[0008] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide an activity recognition device, activity recognition method, and program that have a high degree of freedom in the location of the camera that acquires images and have high recognition accuracy. [Means for solving the problem]

[0009] According to one example of the present disclosure, an activity recognition device that recognizes an activity type of a worker includes a learning unit that generates a trained model by performing machine learning using training data, and a recognition unit that recognizes the activity type using the trained model. The learning unit generates a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming the worker's work site. The learning unit uses data indicating the first inter-frame difference image and labels indicating the activity type assigned to the frames included in the first unit section as training data. The recognition unit generates a second inter-frame difference image based on frames included in a second unit section of a recognition-target video obtained by filming the work site. The recognition unit recognizes the activity type of the worker appearing in the second unit section of the recognition-target video based on the output of the trained model when the second inter-frame difference image is input.

[0010] According to this disclosure, the trained model receives as input inter-frame difference images in a unit section. Generally, inter-frame difference images in a unit section where a worker is performing a task differ significantly from inter-frame difference images in a unit section where the worker is absent or where the worker is waiting to perform a task. Therefore, by inputting a second inter-frame difference image in a second unit section of the recognition target video to the trained model, the task recognition device can accurately recognize the task type of the worker appearing in the second unit section. Furthermore, the trained model does not require input of skeleton node information extracted from the image, as in the technology disclosed in Patent Document 1. Therefore, unlike the technology disclosed in Patent Document 1, there are no restrictions on the installation location of the camera. In other words, there is a high degree of freedom in the installation location of the camera for acquiring the training images and the recognition target images.

[0011] In the above disclosure, the activity recognition device further includes a person detection unit that detects a person from each frame of the recognition target video. The recognition unit recognizes the activity type only for frames in which a person is detected. This disclosure reduces the load on the recognition unit.

[0012] For example, if each of the first and second unit sections includes two consecutive frames, the first inter-frame difference image may be a difference image between the two frames included in the first unit section, and the second inter-frame difference image may be a difference image between the two frames included in the second unit section.

[0013] Alternatively, the first inter-frame difference image may be a difference image between one of two frames included in the first unit section and an interpolated frame inserted between the two frames included in the first unit section, and the second inter-frame difference image may be a difference image between one of two frames included in the second unit section and an interpolated frame inserted between the two frames included in the second unit section.

[0014] Alternatively, when each of the first unit section and the second unit section includes three or more consecutive frames, the first inter-frame difference image may be a difference image between the first frame and the last frame of the three or more frames included in the first unit section, and the second inter-frame difference image may be a difference image between the first frame and the last frame of the three or more frames included in the second unit section.

[0015] Alternatively, when each of the first and second unit intervals includes four or more consecutive frames, the first inter-frame difference image may be a difference image between a frame obtained by weighting the first group of four or more frames included in the first unit interval and a frame obtained by weighting the second group of four or more frames included in the first unit interval. The second inter-frame difference image may be a difference image between a frame obtained by weighting the first group of four or more frames included in the second unit interval and a frame obtained by weighting the second group of four or more frames included in the second unit interval.

[0016] In the above disclosure, the learning unit generates a trained model using few-shot learning. According to this disclosure, the trained model can be generated using a small amount of training data.

[0017] According to an example of the present disclosure, an activity recognition method for recognizing an activity type of a worker includes: generating a trained model by one or more processors performing machine learning using training data; and recognizing the activity type using the trained model by the one or more processors. Generating the trained model includes generating a first inter-frame difference image based on frames included in a first unit section of a training video obtained by filming the worker's work site, and using data indicating the first inter-frame difference image and labels indicating the activity type assigned to the frames included in the first unit section as training data. Recognizing includes generating a second inter-frame difference image based on frames included in a second unit section of a recognition-target video obtained by filming the work site, and recognizing the activity type of the worker appearing in the second unit section of the recognition-target video based on output of the trained model when the second inter-frame difference image is input.

[0018] According to yet another example of the present disclosure, a program causes a computer to execute the above information processing method.

[0019] These disclosures also provide a high degree of freedom in the location of installation of the camera that captures images, and a high degree of accuracy in recognizing tasks. [Effects of the Invention]

[0020] According to the present disclosure, there is a high degree of freedom in the location of installation of a camera that captures images, and the accuracy of task recognition is high. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a diagram illustrating an example of a system to which an activity recognition device according to an embodiment is applied; [Figure 2]10 is a flowchart illustrating an example of the flow of a trained model generation process. [Figure 3] FIG. 10 is a diagram illustrating a method for generating teacher data. [Figure 4] FIG. 10 is a diagram illustrating another example of a method for generating teacher data. [Figure 5] FIG. 10 is a diagram showing a first example of generation of an inter-frame difference image. [Figure 6] FIG. 10 is a diagram showing a second example of generating an inter-frame difference image. [Figure 7] FIG. 10 is a diagram showing a third example of generation of an inter-frame difference image. [Figure 8] FIG. 10 is a diagram showing a fourth example of generation of an inter-frame difference image. [Figure 9] 10 is a flowchart illustrating an example of the flow of a process for recognizing an operation type. [Figure 10] 10 is a flowchart showing another example of the flow of the work type recognition process. [Figure 11] FIG. 10 is a diagram illustrating an example of an error in recognizing a work type. DETAILED DESCRIPTION OF THE INVENTION

[0022] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals and the description thereof will not be repeated.

[0023] §1 Reference form Before describing the embodiments of the present disclosure, a reference embodiment and its problems will be described. Similar to Patent Document 2, for example, an activity recognition device according to the reference embodiment uses a trained model that receives input of video frames obtained by filming the workplace of a worker and outputs the probability that the frame will be classified into each of a plurality of activity types.

[0024] The trained model used by the task recognition device according to the reference embodiment receives video frames directly as input. Therefore, the task recognition device according to the reference embodiment has the advantage of not requiring the preprocessing required to acquire skeleton node information, as described in Patent Document 1. However, the accuracy of task type recognition tends to be low. In particular, videos taken at work sites often include various objects other than workers. This can easily result in low recognition accuracy.

[0025] 11 is a diagram showing an example of a recognition error of a task type. In the example shown in FIG. 11, the trained model is generated using training data showing an image 60 of a worker 13 performing a task of task type "A" and a classification label representing task type "A." Task type "A" represents a task performed by worker 13 on a board 70.

[0026] Image 61 does not include worker 13. However, if image 61 is similar to image 60 except for the presence or absence of worker 13, the trained model may recognize the type of work shown in image 61 as work type "A."

[0027] Image 62 shows a state in which worker 13 is waiting for board 70. However, the state of worker 13 in image 62 is similar to the state of worker 13 in image 60. Therefore, the trained model can recognize the type of work shown in image 62 as work type "A."

[0028] As described above, the trained model according to the reference embodiment, which receives as input frames of a video captured at a work site, has low recognition accuracy. In particular, the trained model according to the reference embodiment may erroneously recognize the task type in frames in which no worker is present or in which the worker is waiting to perform the task.

[0029] In contrast, the task recognition device according to this embodiment can improve the recognition accuracy for frames in which no worker is present or in which the worker is waiting to perform a task, as will be described later.

[0030] §2 Examples of application Fig. 1 is a diagram illustrating a schematic example of a system to which an activity recognition device according to an embodiment is applied. The system 1 illustrated in Fig. 1 includes a production site 10, an equipment management server 20, an activity recognition device 30, and a visualization server 40. Each of the equipment management server 20, the activity recognition device 30, and the visualization server 40 may be configured with one or more computers, and may include a virtual machine or a container built in a cloud environment, or a configuration consisting of at least a part of these.

[0031] One or more pieces of equipment 11 and one or more network cameras 12 are installed at the production site 10. The one or more pieces of equipment 11 include, for example, manufacturing equipment, sensors, conveyance equipment, etc. The one or more pieces of equipment 11 output equipment signals indicating their operating status to the equipment management server 20 at predetermined control cycles.

[0032] One or more network cameras 12 capture images of workers 13 performing work at the production site 10. The one or more network cameras 12 output video data obtained by capturing the images (hereinafter simply referred to as "video") to the task recognition device 30. Each frame of the video may be a color image or a monochrome image.

[0033] The equipment management server 20 receives equipment signals from one or more pieces of equipment 11 and manages the values ​​of one or more variables that indicate the state of the one or more pieces of equipment 11. The equipment management server 20 includes an equipment database 21 that stores the values ​​of the one or more variables for each control period.

[0034] The task recognition device 30 recognizes tasks performed by workers 13 at the production site 10. The task recognition device 30 is configured by a computer with a general-purpose architecture. The task recognition device 30 includes, as its main components, a processor 31, a memory 32, and a user interface 38. The processor 31 includes, for example, a central processing unit (CPU) or a micro processing unit (MPU). The memory 32 includes, for example, a volatile storage device such as a dynamic random access memory (DRAM). The user interface 38 includes a display and an input device. The input device includes, for example, a keyboard, a mouse, or a touchpad.

[0035] The task recognition device 30 further includes a recording service 35, an task recognition application 36, and an annotation tool 37 as programs executed by the processor 31. These programs are stored in a storage device such as a hard disk or a storage medium. Storage media include volatile storage media, nonvolatile storage media, general-purpose semiconductor storage devices such as Compact Flash (CF) or Secure Digital (SD), magnetic storage media such as Flexible Disks, and optical storage media such as Compact Disk Read Only Memory (CD-ROM). Some or all of the functions provided by these programs may be realized by dedicated hardware circuits (e.g., Application Specific Integrated Circuits (ASICs) or Field-Programmable Gate Arrays (FPGAs)). The task recognition device 30 further includes a task recognition database 34.

[0036] The recording service 35 outputs a recording instruction to the network camera 12 and causes the processor 31 to execute an operation of collecting videos obtained by shooting in accordance with the recording instruction. The videos collected by executing the recording service 35 include learning videos for generating teacher data and recognition target videos that are targets for task type recognition.

[0037] The annotation tool 37 causes the processor 31 to perform an operation of assigning a classification label selected from a plurality of classification labels representing task types to each frame of the learning video in response to an input to the user interface 38. The annotation tool 37 causes the processor 31 to perform an operation of generating annotation data indicating the classification labels assigned to each frame of the learning video.

[0038] The task recognition application 36 causes the processor 31 to execute the following processes: generating a trained model, recognizing task types using the trained model, and storing the task type recognition results in the task recognition database 34. When the processor 31 executes the task recognition application 36, a learning unit 311, a recognition unit 312, and a human detection unit 313 are realized.

[0039] In addition to the recognition results of the task types, the task recognition database 34 may store equipment data indicating some or all of the values ​​of one or more variables for each control period stored in the equipment database 21.

[0040] The learning unit 311 generates a trained model by performing machine learning using training data. The training data is generated based on annotation data. Specifically, the learning unit 311 generates a first inter-frame difference image based on frames included in a first unit section of the training video. The learning unit 311 uses data indicating the first inter-frame difference image and classification labels assigned to the frames included in the first unit section as training data.

[0041] In general machine learning, a trained model is generated using a large amount of training data. In contrast, in few-shot learning, a trained model is generated using a small amount of training data. In general, the recognition accuracy of a trained model generated using few-shot learning is lower than the recognition accuracy of a trained model generated using general machine learning. However, as will be described later, the task recognition accuracy of the task recognition device 30 according to this embodiment is high. Therefore, the learning unit 311 may generate a trained model using few-shot learning.

[0042] The recognition unit 312 uses the trained model to recognize the type of work shown in each frame of the recognition target image. Specifically, the recognition unit 312 generates a second inter-frame difference image based on frames included in a second unit section of the recognition target video obtained by filming the work site of the worker 13. The recognition unit 312 recognizes the type of work of the worker shown in the second unit section based on the output of the trained model when the second inter-frame difference image is input. In the following description, when the first unit section and the second unit section are not distinguished from each other, each of the first unit section and the second unit section is simply referred to as a "unit section." When the first inter-frame difference image and the second inter-frame difference image are not distinguished from each other, each of the first inter-frame difference image and the second inter-frame difference image is simply referred to as an "inter-frame difference image."

[0043] The human detection unit 313 detects a human from each frame of the recognition target video. The detection result by the human detection unit 313 can be used to select frames to be recognized as work types.

[0044] The visualization server 40 provides a visualization service 41 that visualizes the recognition results obtained by the task recognition device 30. For example, in response to a request from a user terminal (not shown), the visualization service 41 accesses the task recognition database 34 to acquire data to be visualized (visualization data). The visualization service 41 generates a screen showing the task type recognition results based on the visualization data and outputs the generated screen to the user terminal. This allows the user to check the task type recognition results using the user terminal.

[0045] The processing of a processor operating as a learning unit will be described with reference to Figs. 2 to 8. Fig. 2 is a flowchart showing an example of the flow of processing for generating a trained model. Fig. 3 is a diagram illustrating an example of a method for generating training data. Fig. 4 is a diagram illustrating another example of a method for generating training data. Fig. 5 is a diagram illustrating a first example of generating an inter-frame difference image. Fig. 6 is a diagram illustrating a second example of generating an inter-frame difference image. Fig. 7 is a diagram illustrating a third example of generating an inter-frame difference image. Fig. 8 is a diagram illustrating a fourth example of generating an inter-frame difference image.

[0046] First, in step S1, the processor 31 operating as the learning unit 311 acquires annotation data. As shown in Figures 3 and 4, the annotation data 80 indicates classification labels assigned to each frame of the learning video.

[0047] In the next step S2, the processor 31 generates a first inter-frame difference image based on frames included in a first unit interval of the learning video. The time length of the first unit interval is determined in advance according to the frame rate of the learning video. For example, the first unit interval may have a time length including two consecutive frames. Alternatively, the first unit interval may have a time length including three or more consecutive frames.

[0048] The method of generating training data shown in Fig. 3 and the example of generating an inter-frame difference image shown in Fig. 5 and 6 correspond to a unit section having a time length including two consecutive frames. The method of generating training data shown in Fig. 4 and the example of generating an inter-frame difference image shown in Fig. 7 and 8 correspond to a unit section having a time length including three or more consecutive frames.

[0049] When a unit section having a time length including two consecutive frames is set, processor 31 may generate a difference image between the two frames included in the unit section as an inter-frame difference image. For example, as shown in FIG. 5, processor 31 generates a difference image 52 between the (k-1)th frame 50 and the kth frame 51 of the video as an inter-frame difference image. Then, as shown in FIG. 3, for each first unit section of the training video, processor 31 uses data indicating the first inter-frame difference image and classification labels assigned to the frames included in the first unit section as training data 85. In the example shown in FIG. 3, processor 31 generates training data 85 indicating a difference image "1, 2" between the first and second frames and a classification label "A." Furthermore, processor 31 generates training data 85 indicating a difference image "2, 3" between the second and third frames and a classification label "A." Furthermore, processor 31 generates training data 85 indicating a differential image "N-1,N" between the (N-1)th frame and the Nth frame and a classification label "C."

[0050] Alternatively, as shown in FIG. 6 , the processor 31 may insert an interpolated frame 53 between the (k−1)th frame 50 and the kth frame 51 of the video. The processor 31 may generate the interpolated frame 53 using a known technique. The processor 31 may then generate a difference image 54 between the frame 50 and the interpolated frame 53 as an inter-frame difference image. Alternatively, the processor 31 may generate a difference image between the interpolated frame 53 and the frame 51 as an inter-frame difference image instead of or in addition to the difference image 54. The processor 31 then uses, for each first unit section of the training video, data indicating the generated first inter-frame difference image and the classification labels assigned to the frames included in that first unit section as training data 85.

[0051] When a unit interval having a time length including three or more consecutive frames is set, processor 31 may generate, as an inter-frame difference image, a difference image between the first and last frames in the unit interval. For example, as shown in Fig. 7, processor 31 generates, as an inter-frame difference image, a difference image 56 between the (k-2)th frame 55 and the kth frame 51 of the video. The (k-1)th frame 50 is thinned out and is not used to generate the inter-frame difference image.

[0052] Alternatively, processor 31 may divide the four or more frames included in the unit section into a first half frame group and a second half frame group. Then, processor 31 may generate, as an inter-frame difference image, a difference image between a frame obtained by weighting the first half frame group and a frame obtained by weighting the second half frame group. For example, as shown in FIG. 8 , processor 31 generates, as an inter-frame difference image, a difference image 59 between frame 57a obtained by weighting the (k-5) to (k-3) th frame group 57 included in the unit section and frame 58a obtained by weighting the (k-2) to (k) th frame group 58 included in the unit section.

[0053] Processor 31 uses, as training data 85, data indicating a first inter-frame difference image generated based on three or more frames included in each first unit section of the learning video and classification labels assigned to the frames included in the first unit section. In the example shown in FIG. 4, processor 31 generates training data 85 indicating a difference image "1,p" generated from the 1st to pth frames and a classification label "A". Furthermore, processor 31 generates training data 85 indicating a difference image "N-p+1,N" generated from the N-p+1st to Nth frames and a classification label "C".

[0054] Note that, when different classification labels are assigned to a plurality of frames included in a first unit section, the processor 31 may omit generating the training data 85 for the first unit section.

[0055] In the next step S3, the processor 31 generates a trained model by performing machine learning using the training data 85. The trained model receives an input of a difference image between two consecutive frames and outputs the probability that the difference image is classified into each of a plurality of classification labels. The trained model may be generated using few-shot learning. This reduces the amount of training data 85 required to generate the trained model.

[0056] In the next step S4, the processor 31 outputs the generated trained model. Specifically, the processor 31 stores the trained model in the memory 32 or a storage device (not shown). After step S4, the processing of the processor 31 operating as the training unit 311 ends.

[0057] The work type recognition process will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the flow of the work type recognition process.

[0058] First, in step S11, the processor 31 operating as the recognition unit 312 reads out a trained model. In step S12, the processor 31 acquires a video to be recognized. The video to be recognized is output from the network camera 12. The order of steps S11 and S12 is not particularly limited.

[0059] In the next step S13, processor 31 generates a second inter-frame difference image based on frames included in a second unit section of the recognition target video. The method for generating the second inter-frame difference image is the same as the method for generating the first inter-frame difference image. Note that if the frame rates of the recognition target video and the learning video are the same, the first unit section and the second unit section have the same time length, and the first inter-frame difference image and the second inter-frame difference image are generated by the same method. For example, if the first inter-frame difference image is generated according to the first generation example shown in FIG. 5, the second inter-frame difference image is also generated according to the first generation example.

[0060] In the next step S14, the processor 31 recognizes the task type of the worker 13 appearing in the second unit section corresponding to the second frame difference image, based on the output of the trained model when the second frame difference image is input. As described above, the trained model receives the input frame difference image and outputs the probability that the frame difference image will be classified into each of multiple classification labels. Therefore, the processor 31 recognizes the task type represented by the classification label corresponding to the highest probability output from the trained model as the task type of the worker 13 appearing in the second unit section.

[0061] Steps S13 and S14 are performed for all second unit segments included in the authentication target video. When steps S13 and S14 have been performed for all second unit segments, the process proceeds to step S15.

[0062] In step S15, processor 31 stores the recognition result for each second unit section in task recognition database 34. After step S15, the process of processor 31 operating as recognition unit 312 ends.

[0063] While the worker 13 is working, the worker's 13 hands and the object being worked on usually move. Therefore, as shown in FIG. 5 , the difference image 52 shows high brightness in moving parts. In contrast, no movement is observed in the video during unit sections where the worker 13 is absent. Furthermore, no movement is usually observed in the video during unit sections where the worker 13 is waiting. Therefore, the frame-to-frame difference image during a unit section where the worker 13 is working is significantly different from the frame-to-frame difference image during a unit section where the worker 13 is absent or waiting. Therefore, by using a trained model generated by performing machine learning using training data indicating the frame-to-frame difference image and the classification label, the task recognition device 30 can accurately recognize the task type depicted in the unit section of the recognition target video. In other words, erroneous recognition of frames included in unit sections where the worker 13 is absent or waiting as frames showing the worker 13 working is suppressed.

[0064] Furthermore, the trained model according to this embodiment receives input of an inter-frame difference image, and does not require input of skeleton node information extracted from the image as in the technology disclosed in Patent Document 1. Therefore, unlike the technology disclosed in Patent Document 1, there are no restrictions on where the camera can be installed, and there is a high degree of freedom in where the network camera 12 can be installed.

[0065] Fig. 10 is a flowchart showing another example of the flow of the task type recognition process. The flow shown in Fig. 10 differs from the flow shown in Fig. 9 in that step S20 is included between step S12 and step S13.

[0066] In step S20, processor 31 operating as human detection unit 313 detects a human from each frame of the video to be recognized. Processor 31 may use known object recognition technology to determine whether a human is present in each frame. Processor 31 excludes frames in which a human is not detected from the processing targets of step S13 and thereafter. In other words, processor 31 executes the processing of step S13 and thereafter only for frames in which a human is detected. As a result, processor 31 operating as recognition unit 312 recognizes the task type only for frames in which a human is detected. As a result, the load on processor 31 is reduced.

[0067] §3 Supplementary Note As described above, the present embodiment includes the following disclosures.

[0068] (Configuration 1) A task recognition device (30) that recognizes a task type of a worker (13), A learning unit (31, 311) that generates a trained model by performing machine learning using training data (85); A recognition unit (31, 312) that recognizes the work type using the trained model, The learning unit (31, 311) generating a first inter-frame difference image based on frames included in a first unit section of a learning video obtained by filming the work site of the worker; data indicating the first inter-frame difference image and a label indicating the task type assigned to the frame included in the first unit section are used as the training data (85); The recognition unit (31, 312) generating a second inter-frame difference image based on frames included in a second unit section of the recognition target video obtained by capturing an image of the work site; A task recognition device (30) that recognizes the task type of the worker appearing in the second unit section based on the output of the trained model when the second inter-frame difference image is input.

[0069] (Configuration 2) The recognition target video further includes a person detection unit (31, 313) that detects a person from each frame of the recognition target video, The task recognition device (30) according to configuration 1, wherein the recognition unit (31, 312) recognizes the task type only for frames in which a person is detected.

[0070] (Configuration 3) each of the first unit interval and the second unit interval includes two consecutive frames; the first inter-frame difference image is a difference image between the two frames included in the first unit section, 3. The task recognition device (30) according to configuration 1 or 2, wherein the second inter-frame difference image is a difference image between the two frames included in the second unit section.

[0071] (Configuration 4) each of the first unit interval and the second unit interval includes two consecutive frames; the first inter-frame difference image is a difference image between one of the two frames included in the first unit section and an interpolated frame inserted between the two frames included in the first unit section, The task recognition device (30) according to configuration 1 or 2, wherein the second inter-frame difference image is a difference image between one of the two frames included in the second unit section and an interpolated frame inserted between the two frames included in the second unit section.

[0072] (Configuration 5) each of the first unit interval and the second unit interval includes three or more consecutive frames; the first inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the first unit section, 3. The task recognition device (30) according to configuration 1 or 2, wherein the second inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the second unit section.

[0073] (Configuration 6) each of the first unit interval and the second unit interval includes four or more consecutive frames; the first inter-frame difference image is a difference image between a frame obtained by weighting a first half of a group of the four or more frames included in the first unit section and a frame obtained by weighting a second half of a group of the four or more frames included in the first unit section, The task recognition device (30) according to configuration 1 or 2, wherein the second inter-frame difference image is a difference image between a frame obtained by taking a weighted average of a first half group of the four or more frames included in the second unit section and a frame obtained by taking a weighted average of a second half group of the four or more frames included in the second unit section.

[0074] (Configuration 7) 7. The task recognition device (30) according to any one of configurations 1 to 6, wherein the learning unit (31, 311) generates the trained model using few-shot learning.

[0075] (Configuration 8) A task recognition method for recognizing a task type of a worker (13), comprising: One or more processors (31) perform machine learning using training data (85) to generate a trained model (S1 to S4); The one or more processors (31) recognize the work type using the trained model (S11 to S15), Generating the trained model (S1 to S4) generating a first inter-frame difference image based on frames included in a first unit section of a learning video obtained by filming the work site of the worker (S2); using data indicating the first inter-frame difference image and a label indicating the task type assigned to the frame included in the first unit section as the training data (85) (S3); The recognizing (S11 to S15) generating a second inter-frame difference image based on frames included in a second unit section of the recognition target video obtained by shooting the work site (S13); and recognizing (S14) the type of work of the worker appearing in the second unit section of the video to be recognized, based on the output of the trained model when the second inter-frame difference image is input.

[0076] (Configuration 9) A program (36) for causing a computer (31) to execute a task recognition method for recognizing a task type of a worker (13), the task recognition method comprising: Generating a trained model by performing machine learning using training data (85) (S1 to S4); Recognizing the work type using the trained model (S11 to S15), Generating the trained model (S1 to S4) generating a first inter-frame difference image based on frames included in a first unit section of a learning video obtained by filming the work site of the worker (S2); using, as the training data, data indicating the first inter-frame difference image and a label indicating the task type assigned to the frame included in the first unit section (S3); The recognizing (S11 to S15) generating a second inter-frame difference image based on frames included in a second unit section of the recognition target video obtained by shooting the work site (S13); A program (36) including recognizing (S14) the work type of the worker appearing in the second unit section of the recognition target video based on the output of the trained model when the second inter-frame difference image is input.

[0077] Although the embodiments of the present invention have been described, the embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0078] 1 System, 10 Production site, 11 Equipment, 12 Network camera, 13 Worker, 20 Equipment management server, 21 Equipment database, 30 Task recognition device, 31 Processor, 32 Memory, 34 Task recognition database, 35 Recording service, 36 Task recognition application, 37 Annotation tool, 38 User interface, 40 Visualization server, 41 Visualization service, 50, 51, 55, 57a, 58a Frame, 52, 54, 56, 59 Difference image, 53 Interpolated frame, 57, 58 Frame group, 60, 61, 62 Image, 70 Board, 80 Annotation data, 85 Training data, 311 Learning unit, 312 Recognition unit, 313 Human detection unit.

Claims

1. A task recognition device that recognizes a task type of a worker, a learning unit that generates a trained model by performing machine learning using training data; A recognition unit that recognizes the work type using the trained model, The learning unit generating a first inter-frame difference image based on frames included in a first unit section of a learning video obtained by filming the work site of the worker; data indicating the first inter-frame difference image and a label indicating the task type assigned to the frame included in the first unit section are used as the training data; The recognition unit generating a second inter-frame difference image based on frames included in a second unit section of the recognition target video obtained by shooting the work site; A task recognition device that recognizes the task type of the worker appearing in the second unit section based on the output of the trained model when the second inter-frame difference image is input.

2. further comprising a person detection unit that detects a person from each frame of the recognition target video; The task recognition device according to claim 1 , wherein the recognition unit recognizes the task type only for frames in which a person is detected.

3. each of the first unit interval and the second unit interval includes two consecutive frames; the first inter-frame difference image is a difference image between the two frames included in the first unit section, The task recognition apparatus according to claim 1 , wherein the second inter-frame difference image is a difference image between the two frames included in the second unit section.

4. each of the first unit interval and the second unit interval includes two consecutive frames; the first inter-frame difference image is a difference image between one of the two frames included in the first unit section and an interpolated frame inserted between the two frames included in the first unit section, 3. The task recognition device according to claim 1, wherein the second inter-frame difference image is a difference image between one of the two frames included in the second unit section and an interpolated frame inserted between the two frames included in the second unit section.

5. each of the first unit interval and the second unit interval includes three or more consecutive frames; the first inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the first unit section, The task recognition device according to claim 1 , wherein the second inter-frame difference image is a difference image between a first frame and a last frame of the three or more frames included in the second unit section.

6. each of the first unit interval and the second unit interval includes four or more consecutive frames; the first inter-frame difference image is a difference image between a frame obtained by taking a weighted average of a first half frame group of the four or more frames included in the first unit section and a frame obtained by taking a weighted average of a second half frame group of the four or more frames included in the first unit section, 3. The task recognition device according to claim 1, wherein the second inter-frame difference image is a difference image between a frame obtained by taking a weighted average of a first half of the four or more frames included in the second unit interval and a frame obtained by taking a weighted average of a second half of the four or more frames included in the second unit interval.

7. The task recognition device according to claim 1 , wherein the learning unit generates the trained model using few-shot learning.

8. A task recognition method for recognizing a task type of a worker, comprising: One or more processors perform machine learning using training data to generate a trained model; The one or more processors recognize the task type using the trained model; Generating the trained model includes: generating a first inter-frame difference image based on frames included in a first unit section of a learning video obtained by filming the work site of the worker; data indicating the first inter-frame difference image and a label indicating the task type assigned to the frame included in the first unit section are used as the training data; The recognizing generating a second inter-frame difference image based on frames included in a second unit section of the recognition target video obtained by shooting the work site; and recognizing the task type of the worker appearing in the second unit section of the video to be recognized based on the output of the trained model when the second inter-frame difference image is input.

9. A program for causing a computer to execute a task recognition method for recognizing a task type of a worker, the task recognition method comprising: Generating a trained model by performing machine learning using training data; Recognizing the work type using the trained model; Generating the trained model includes: generating a first inter-frame difference image based on frames included in a first unit section of a learning video obtained by filming the work site of the worker; data indicating the first inter-frame difference image and a label indicating the task type assigned to the frame included in the first unit section are used as the training data; The recognizing generating a second inter-frame difference image based on frames included in a second unit section of the recognition target video obtained by shooting the work site; and recognizing the work type of the worker appearing in the second unit section of the video to be recognized based on the output of the trained model when the second inter-frame difference image is input.

Images