Model learning device, behavior recognition device, model learning method, and model learning program

Abstract

To provide a model learning device, a model learning method, a model learning program, and a behavior recognition device capable of improving a model behavior recognition accuracy learned by a few-shot learning.SOLUTION: A behavior recognition device 1 operating as a model learning device comprises an input data acquisition part to acquire learning data generated by user devices including captured images of a person manipulating an object, an extraction part to extract the support and query used for learning based on learning data, a frame selection part to select uniformly predetermined number of frames from the captured images, a convergence condition determination part to determine whether or not the person and the object interact in the selected frame, a frame replacement part to replace the frame determined not to interact with the frame in which the person and the object interact based on the prescribed criteria, and a model update part to update the model parameter based on the replaced prescribed number of the frames and on the uniformly selected prescribed number the frames.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a model learning device, a behavior recognition device, a model learning method, and a model learning program. [Background technology]

[0002] Due to a labor shortage, digitalization of operations is required in various industries. When digitalizing operations, it is necessary to understand how the operations to be digitized are being performed. By understanding how operations are being performed, it becomes possible to analyze and improve operations, for example, by visualizing the behavior of store employees to analyze and streamline operations, or by visualizing the behavior of medical professionals to understand which are medical procedures and which are not. However, observing people's behavior using cameras, etc., takes a huge amount of time.

[0003] For example, there is a technology that uses behavior recognition technology to automatically capture human behavior from a camera. Using this behavior recognition technology, it is possible to visualize a person's actual behavior and use it to analyze and improve business operations. However, there is a problem in that it is costly to define the behavior to be recognized (defining behavior labels). Therefore, in practical operations, it is useful to narrow down the behaviors to be analyzed to a certain extent in advance and learn certain behavior labels to detect those behaviors with a small amount of training data.

[0004] For example, there is a technology that uses a learning method called few-shot learning to enable a model to learn a certain class of behavior using only a few video images (see, for example, Non-Patent Document 1). Here, the class of behavior is a categorization of actions such as walking and touching, and may be a more abstract category such as climbing a stepladder. When learning a model, the classes are represented by action labels such as walking and touching. Few-shot learning generates classification tasks called episodes to construct learning data. Specifically, suppose there is learning data with a large number of classes, and five classes are extracted from it. If three samples of each class are held as learning data, the support will be 5 way 3 shots (5 classes, 3 samples). At that time, when the query holds one video of each of two classes out of the five classes, the task of few-shot learning is to learn with the support and accurately classify the videos of the two classes. In addition, the combination of this support and query is called an episode. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Jiahao Wang, Yunhong Wang, Sheng Liu, Annan Li, “Few-shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning”, ACMMM2021. Summary of the Invention [Problem to be solved by the invention]

[0006] When recognizing an action that takes a certain amount of time (hereinafter referred to as an activity), it is common to sample and use the input video data rather than using it as is.

[0007] For example, since few-shot learning learns from a small amount of data, depending on the sampling method, there may be a problem that there is not enough data to classify the behavior, and the accuracy of behavior recognition may decrease. Therefore, it is necessary to sample while leaving the data necessary for use in few-shot learning when recognizing activities.

[0008] This invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a technology that makes it possible to improve the action recognition accuracy of a model trained using few-shot learning by performing sampling that takes into account the interaction between the hand of a person performing an action and an object. [Means for solving the problem]

[0009] In order to solve the above problem, a model learning device of one embodiment of the present invention includes a learning data acquisition unit that acquires learning data including captured images of a person manipulating an object; an extraction unit that extracts support and queries to be used for learning based on the learning data; a frame selection unit that uniformly selects a predetermined number of frames from the captured images; a determination unit that determines whether the person and the object are interacting in the uniformly selected frames; a frame replacement unit that replaces frames determined to not be interacting with frames in which the person and the object are interacting based on a predetermined criterion; and a model update unit that updates model parameters based on the replaced predetermined number of frames and the uniformly selected predetermined number of frames. Effect of the Invention

[0010] According to one aspect of the present invention, by performing sampling that takes into account the interaction between the hand of a person performing an action and an object, it is possible to improve the action recognition accuracy of a model trained using few-shot learning. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of a behavior recognition device according to this embodiment. [Diagram 2] FIG. 2 is a block diagram showing the software configuration of the behavior recognition device according to this embodiment in association with the hardware configuration shown in FIG. [Diagram 3] FIG. 3 is a flowchart showing an example in which the behavior recognition apparatus estimates and outputs model parameters. [Figure 4] FIG. 4 is a flowchart showing an example for explaining the process of step S13 in more detail. [Diagram 5] FIG. 5 is a diagram showing an example of an operation for selecting a frame. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Next, an embodiment of the present invention will be described below with reference to the drawings. In the following, elements that are the same as or similar to elements already described will be given the same or similar reference numerals, and duplicated descriptions will basically be omitted. For example, when there are multiple identical or similar elements, a common reference numeral may be used to describe each element without distinguishing between them, and a subnumber may be used in addition to the common reference numeral to describe each element with distinction between them.

[0013] [Embodiment] (composition) FIG. 1 is a block diagram showing an example of a hardware configuration of a behavior recognition device 1 according to this embodiment. The behavior recognition device 1 is a computer that analyzes input data and generates and outputs output data. For example, the behavior recognition device 1 is installed at an arbitrary location set by a user who manages the behavior recognition device 1. The behavior recognition device 1 also operates as a model learning device.

[0014] As shown in Fig. 1, the behavior recognition device 1 includes a control unit 10, a program storage unit 20, a data storage unit 30, a communication interface 40, and an input / output interface 50. The control unit 10, the program storage unit 20, the data storage unit 30, the communication interface 40, and the input / output interface 50 are communicatively connected to each other via a bus. Furthermore, the communication interface 40 may be communicatively connected to an external device via a network. Moreover, the input / output interface 50 is communicatively connected to an input device 2, an output device 3, and a user device 4.

[0015] The control unit 10 controls the behavior recognition device 1. The control unit 10 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 10 may be an integrated circuit capable of executing various programs.

[0016] The program storage unit 20 can use, as a storage medium, a combination of a non-volatile memory that can be written and read at any time, such as an EPROM (Erasable Programmable Read Only Memory), a HDD (Hard Disk Drive), or an SSD (Solid State Drive), and a non-volatile memory such as a ROM (Read Only Memory). The program storage unit 20 stores programs necessary for executing various processes. That is, the control unit 10 can realize various controls and operations by reading and executing the programs stored in the program storage unit 20.

[0017] The data storage unit 30 is a storage that uses a combination of a non-volatile memory that can be written to and read from at any time, such as a HDD or a memory card, and a volatile memory such as a RAM (Random Access Memory), as a storage medium. The data storage unit 30 is used to store data acquired and generated in the process of the control unit 10 executing a program and performing various processes.

[0018] The communication interface 40 includes one or more wired or wireless communication modules. For example, the communication interface 40 includes a communication module that connects to an external device via a network in a wired or wireless manner. The communication interface 40 may include a wireless communication module that connects wirelessly to an external device such as a Wi-Fi access point or a base station. Furthermore, the communication interface 40 may include a wireless communication module for wirelessly connecting to an external device using a short-range wireless technology. That is, the communication interface 40 may be a general communication interface as long as it can communicate with an external device under the control of the control unit 10 and transmit and receive various information.

[0019] The input / output interface 50 is connected to the input device 2, the output device 3, the user device 4, and the like. The input / output interface 50 is an interface that enables transmission and reception of information between the input device 2, the output device 3, and the user device 4. The input / output interface 50 may be integrated with the communication interface 40. For example, the behavior recognition device 1 and at least one of the input device 2, the output device 3, and the user device 4 are wirelessly connected using a short-range wireless technology or the like, and information may be transmitted and received using the short-range wireless technology.

[0020] The input device 2 may include, for example, a keyboard, a pointing device, or the like, through which a user inputs various information to the behavior recognition device 1. The input device 2 may also include a reader for reading data to be stored in the program storage unit 20 or the data storage unit 30 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0021] The output device 3 includes a display or the like that displays the results of calculations performed by the control unit 10, action labels, etc. The output device 3 also includes a printer or the like that prints out the information displayed on the display.

[0022] The user device 4 is a device including a camera that captures the movement of a user (a person performing an action). That is, the user device 4 is a device used when generating learning data to be input to the behavior recognition device 1. Details of the learning data will be described later. The user device 4 may be any device that can acquire data to be used for learning data and output the data to the behavior recognition device 1.

[0023] FIG. 2 is a block diagram showing the software configuration of the behavior recognition device 1 according to this embodiment in association with the hardware configuration shown in FIG. The control unit 10 includes an input data acquisition unit 101, a learning dataset generation unit 102, a model learning unit 103, a behavior recognition unit 104, and an output control unit 105.

[0024] The input data acquisition unit 101 acquires learning data. Here, the learning data includes a captured image of a user (person) operating an object. For example, the captured image may be an image captured by a camera (for example, the user device 4) worn by the user (person). That is, the captured image may be an image from a first-person perspective. Therefore, the captured image may be an image of the user's (person's) hand and an object operated by the user's hand. The captured image may be a video of the same operation performed multiple times, that is, multiple captured images. The captured image may be an image from a third-person perspective decoded from an image acquired from a surveillance camera. Therefore, the captured image may be an image of the user's (person's) hand at a desk and an object operated by the user's hand. The captured image may be a video of the same operation performed multiple times, that is, multiple captured images, like the first-person perspective image. The learning data also includes an action label paired with the captured image. The action label may be paired for each captured image or for multiple captured images. The input data acquisition unit 101 outputs the acquired learning data to the learning dataset generation unit 102.

[0025] The training dataset generation unit 102 generates a set of support and query (training dataset) from the training data. A detailed method of generating a set of support and query from the training data will be described later. The training dataset generation unit 102 stores the generated training dataset in the training dataset storage unit 301.

[0026] The model learning unit 103 includes an extraction unit 1031, a frame selection unit 1032, a frame replacement unit 1033, a model update unit 1034, and a convergence condition determination unit 1035. The model learning unit 103 is a learning unit that performs model learning (updating parameters of the model) based on a learning dataset stored in a learning dataset storage unit 301 described later.

[0027] The extraction unit 1031 extracts support and queries for N way K shots. The extraction unit 1031 acquires a learning dataset stored in the learning dataset storage unit 301. Then, the extraction unit 1031 extracts support for N classes and K samples, and a query for evaluation of the same class as the extracted N classes from the learning dataset. Here, a model that can distinguish queries with only a small amount of learning data can be constructed by having the model learn which class sample the query is similar to from a small amount of learning data such as N=5 and K=3. At this time, the model is trained in advance with large-scale learning data other than the learning data, and can distinguish captured images to a certain extent, but by performing the above-mentioned learning, it becomes possible to distinguish queries with only a small amount of learning data.

[0028] The frame selection unit 1032 selects a section where there is interaction between the hand and the object. For example, the frame selection unit 1032 checks frames where there is interaction between the hand and the object in each of the captured images of the support and query extracted by the extraction unit 1031, and selects a section of the captured image where the hand (person) and the object interact. Note that a specific method for selecting a section of the captured image where the person and the object interact will be described later.

[0029] Furthermore, the frame selection unit 1032 uniformly selects L frames from a section where there is interaction between the hand (person) and the object, where L is any positive integer. That is, the frame selection unit 1032 uniformly selects a predetermined number (L frames) of frames from a section where there is interaction between the person and the object.

[0030] The frame replacement unit 1033 replaces necessary frames from the uniformly selected frames with frames selected based on a predetermined criterion. The frame replacement unit 1033 checks whether the L frames uniformly selected by the frame selection unit 1032 are frames with interaction between a hand and an object. If the result of the check is that the frame does not have interaction between a hand and an object, the frame replacement unit 1033 checks frames before and after the frame in question, and replaces the frame with the frame with interaction between a hand and an object that is closest in time.

[0031] The model update unit 1034 calculates the loss using the replaced L (predetermined number) frames and the uniformly selected L (predetermined number) frames, and updates the model. The model update method will be described later in detail.

[0032] The convergence condition determination unit 1035 determines whether the updated model parameters satisfy the convergence condition. For example, if the convergence condition is not satisfied, the system returns to learning using other learning data. The convergence condition here is that the loss is equal to or less than a certain threshold, or the number of updates of the model parameters exceeds a certain number, etc.

[0033] The data storage unit 30 includes a learning dataset storage unit 301 and a model storage unit 302. The learning dataset storage unit 301 is used to store a learning dataset generated by the learning dataset generation unit 102, etc. The learning dataset storage unit 301 may also store a neural network model capable of image recognition that has been trained in advance. The model storage unit 302 is used to store parameters of the model trained by the model training unit 103, etc.

[0034] [Operation] FIG. 3 is a flowchart showing an example in which the behavior recognition device 1 estimates and outputs model parameters. The control unit 10 of the behavior recognition device 1 reads out and executes the program stored in the program storage unit 20, thereby realizing the operation of this flowchart.

[0035] This operation starts when an instruction is given by a user who manages the behavior recognition apparatus 1 or when a captured image is received from the user device 4.

[0036] In step S11, the input data acquisition unit 101 acquires learning data. For simplicity, in this embodiment, a case will be described in which the learning data is a first-person perspective image. However, the present embodiment is not limited to this. For example, the learning data may be any captured image of a person and the person manipulating an object.

[0037] For example, the input data acquisition unit 101 acquires a plurality of captured images (learning data). The input data acquisition unit 101 outputs the acquired learning data to the learning data set generation unit 102.

[0038] In step S12, the learning dataset generation unit 102 generates a set of support and queries (learning dataset) from the learning data. For example, for one model learning process, N classes are selected from the captured image dataset, and K training videos (support set) and M test videos (queries) are extracted for each class, and this is generated as one set. Here, N, K, and M may be any positive integers. That is, the learning dataset generation unit 102 generates a set of support sets and queries the number of times required to learn the model. Specifically, consider a case where the learning dataset has 100 classes, and each class has 1000 training videos and 100 test videos. What we want to evaluate here is whether a model that has been pre-trained from a large amount of learning data can learn and distinguish classes that the model has never learned from a small amount of learning data. Therefore, in order to build a model that can learn with a small amount of learning data, N classes (N is about 1 to 5) are randomly sampled from 100 classes of a learning dataset that are not included in the large-scale learning data used in pre-learning, and K training videos (K is about 1 to 5) are randomly sampled from 1000 samples of each class as learning data. In addition, M, the number of samples to be evaluated for the evaluation query, are randomly sampled from 100 test videos of each of the N classes. All queries may be evaluated without sampling. A combination of the randomly sampled class, training video, and query becomes learning data in few-shot learning. Sampling may not be random sampling, but may be performed according to a certain standard. There is also a method of creating a large number of such combinations, grouping them into a unit called an episode, and learning multiple episodes. In other words, there are multiple sets of combinations of classes, training videos, and queries that have been randomly sampled in large quantities. The learning dataset generation unit 102 stores the generated learning dataset in the learning dataset storage unit 301.

[0039] In step S13, the model learning unit 103 learns a model based on the learning data set.

[0040] FIG. 4 is a flowchart showing an example for explaining the process of step S13 in more detail. In step S21, the extraction unit 1031 extracts support and queries for N way K shots. The extraction unit 1031 acquires a learning dataset stored in the learning dataset storage unit 301. Then, the extraction unit 1031 extracts support for N classes and K samples, and evaluation queries for the same classes as the extracted N classes, from the learning dataset.

[0041] In step S22, the frame selection unit 1032 selects a section in which there is interaction between the hand and the object. For example, the frame selection unit 1032 checks frames in which there is interaction between the hand and the object in each of the photographed images of the support and query extracted by the extraction unit 1031, and selects a section in which the photographed image shows interaction between the hand (person) and the object.

[0042] FIG. 5 is a diagram showing an example of an operation for selecting a frame. In FIG. 5, the captured image is represented by a rectangle whose vertical axis represents the frame height and whose horizontal axis represents the time axis. First, FIG. 5(a) shows an example in which the frame selection unit 1032 selects a section in which a person interacts with an object. As shown in FIG. 5(a), the frame selection unit 1032 checks frames in which a hand interacts with an object from the captured image (frames shown by dots in FIG. 5(b)). Here, a frame in which a hand interacts with an object refers to a frame in which a hand and an object that needs to be operated by the hand are detected using hand detection and object detection, for example, and both are present. Then, the frame selection unit 1032 selects a section in which a hand interacts with an object.

[0043] Here, step S22 may be skipped, i.e., the frame selection unit 1032 does not need to check the section of the captured image in which a person interacts with an object.

[0044] Returning to FIG. 4, in step S23, the frame selection unit 1032 uniformly selects L frames from the section where there is interaction between the hand and the object. Here, L is any positive integer. That is, the frame selection unit 1032 uniformly selects a predetermined number (L frames) of frames from the section where a person and an object interact. With reference to FIG. 5(c), frames are uniformly selected from frames in the section where there is interaction between the hand and the object (i.e., the section where there is interaction between the hand and the object) after deleting the section where there is no interaction between the hand and the object (the selected frames are the frames indicated by the thick black lines).

[0045] In step S24, the frame replacement unit 1033 replaces necessary frames from among the uniformly selected frames with frames selected based on a predetermined criterion. The frame replacement unit 1033 checks whether the L frames uniformly selected by the frame selection unit 1032 are frames with hand-object interaction. If the result of the check is that the frame is without hand-object interaction, the frame replacement unit 1033 checks frames before and after the frame in question, and replaces the frame with the frame with hand-object interaction that is closest in time. That is, the frame replacement unit 1033 replaces the frame without hand-object interaction with a frame with hand-object interaction that is a predetermined criterion (closest in time to the frame in question).

[0046] If the result of the check is that the frame is one in which the hand interacts with the object, the frame is used as is. Therefore, the uniformly selected frame after replacement is selected as shown in (d) of Figure 5.

[0047] In step S25, the model update unit 1034 calculates the loss using the replaced L (predetermined number) frames and the uniformly selected L (predetermined number) frames, and updates the model. The reason for using the uniformly selected frames is to supplement the context information of the entire video that may be lost when the frames are limited to only those in which the hand interacts with the object.

[0048] As described above, the model update unit 1034 updates (learns) the model using frames with interaction between the hand and object (replaced frames + original frames with interaction between the hand and object) and uniformly selected frames. Therefore, it is important to have a model mechanism that expresses the features of each of the frames with interaction between the hand and object and the uniformly selected frames, and takes into account these features as a whole. Here, in a certain captured image query in a set with a support to be learned, a feature (first feature) extracted from a frame with interaction between the hand and object based on a neural network model capable of image recognition pre-trained with a large-scale image recognition dataset or the like is X, and a feature (second feature) extracted from the uniformly selected frames is Y. The neural network model capable of image recognition may be a model stored in the learning dataset storage unit 301. Here, the number of dimensions of the feature is L×D, since it is converted into a D-dimensional feature vector for each frame. Then, using the attention mechanism disclosed in Reference 1 (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, “Attention Is All You Need”, NIPS2017.), when X is a query, the relationship between X and Y is expressed as the following formula (first formula).

[0049] A = softmax(X q Y k / √D)Y v +X Here, softmax represents the softmax function. Also, here, X q =W q X,Y k =W k Y,Y v =W v W represents the fully connected layer, and W q ,W k , and W vis an independent weight. The model update unit 1034 obtains X' (the third feature) by applying the above-mentioned neural network f composed of three fully connected layers according to the following formula:

[0050] X' = ​​f(A) + A (1) In addition, by executing the same operation with the roles of X and Y swapped, we obtain Y' (the fourth feature). In other words, we use Y as a query and apply a neural network to the equation that represents the relationship between Y and X using the same attention mechanism to obtain Y'. In this case, W q ,W k , and W v do not share weights.

[0051] Next, the model update unit 1034 calculates the loss using the calculated X' and Y' (the third feature and the fourth feature) and learns the weights including the neural network model of image recognition used for feature extraction. This enables learning that takes into account the interaction between the hand and the object and the context information of the entire captured image query. Here, the loss learning is performed using a method such as the Temporal-Relational CrossTransformer disclosed in, for example, Literature 2 (Thatipelli, A., Narayan, S., Khan, S., Anwer, RM, Khan, FS and Ghanem, B. "Spatio-Temporal Relation Modeling for Few-Shot Action Recognition", CVPR2022.).

[0052] In step S26, the convergence condition determination unit 1035 determines whether the updated model parameters satisfy the convergence condition. If the convergence condition is satisfied, the process in step S13 is terminated. On the other hand, if the convergence condition is not satisfied, the process returns to learning using other learning data. That is, the process returns to step S21. The convergence condition here is that the loss is equal to or less than a certain threshold, or the number of updates of the model parameters exceeds a certain number, etc.

[0053] By learning the model in step S13 (steps S21 to S26), i.e., by updating the model (parameters) using frames in which there is interaction between the hand and an object and frames selected uniformly, shortcut learning, such as recognizing that a person is climbing a stepladder simply because a frame showing a stepladder is input, can be prevented, and the accuracy of learning can be improved.

[0054] Returning to FIG. 3, in step S14, the model learning unit 103 stores the learned model (parameters) in the model storage unit 302.

[0055] The above processing may be performed as an independent process by, for example, the behavior recognition device 1 operating as a model learning device. That is, the behavior recognition device 1 may perform only the processes of steps S11 to S14 as a model learning device. That is, the processing may be performed as a process separate from the processes (inference process) of steps S15 to S17 described below. In addition, although an example in which the above processing is performed by the behavior recognition device 1 has been described in the present embodiment, the processing may be performed by other devices. Then, the behavior recognition device 1 may store parameters of the learned model in the model storage unit 302.

[0056] In step S15, the behavior recognition unit 104 acquires inference data. The inference data includes, for example, a captured image of a person and an object operation used in learning the model and a captured image similar to this image. For example, if the captured image of a person and an object operation is a captured image of a person climbing a ladder, the inference data includes a captured image of a person climbing a ladder and a captured image of a person climbing a stepladder. The behavior recognition unit 104 acquires the trained model stored in the model storage unit 302.

[0057] In step S16, the behavior recognition unit 104 generates a behavior label using the inference data and the trained model as input. The behavior label generation method may be a general method, so detailed explanations will be omitted. The behavior recognition unit 104 outputs the generated behavior label to the output control unit 105.

[0058] In step S17, the output control unit 105 controls the output device 3 to display the activity label on a display or the like. The user can check, from the activity label, whether the captured image of the inference data is correctly inferred by the model.

[0059] [Effects] According to the embodiment, by performing sampling that takes into account the interaction between the hand of a person performing an action and an object, it is possible to improve the action recognition accuracy of a model trained by few-shot learning.

[0060] [Other embodiments] Furthermore, in the behavior recognition device 1 in the above embodiment, the operations of the components can be constructed as a program, and the program can be installed and executed in the user device 4 or a computer used as an external device.

[0061] The method described in the above embodiment can be stored as a program (software means) that can be executed by a calculator (computer) in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium side also includes a setting program that configures the software means (including not only execution programs but also tables and data structures) that the calculator executes in the computer. The computer that realizes this device reads the program stored in the storage medium, and in some cases, configures the software means by the setting program, and executes the above-mentioned processing by controlling the operation of the software means. The storage medium referred to in this specification is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory provided inside the computer or in a device connected via a network.

[0062] In short, this invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. Moreover, each embodiment may be implemented in combination as appropriate as possible, in which case the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes inventions at various stages, and various inventions can be extracted by appropriate combinations of the disclosed constituent elements. [Explanation of symbols]

[0063] 1…Action recognition device 10...Control section 101...Input data acquisition unit 102...Learning dataset generation unit 103…Model Learning Section 1031...Extraction part 1032…Frame selection section 1033…Frame replacement section 1034…Model Update Section 1035...Convergence condition determination unit 104…Behavior recognition unit 105...Output control unit 20...Program memory section 30...Data storage unit 301...Learning data set storage unit 302…Model storage section 40...Communication interface 50... Input / Output Interface 2. Input device 3. Output device 4. User Device

Claims

1. A learning data acquisition unit that acquires learning data including a captured image of a person manipulating an object; an extraction unit that extracts support and queries to be used in learning based on the learning data; a frame selection unit that uniformly selects a predetermined number of frames from the captured images; a determination unit for determining whether the person and an object are interacting in the uniformly selected frames; a frame replacement unit that replaces the frame determined not to be an interaction with a frame in which the person and the object are interacting based on a predetermined criterion; a model updating unit that updates model parameters based on the replaced predetermined number of frames and the uniformly selected predetermined number of frames; A model learning device comprising:

2. 2. The model learning device according to claim 1, wherein the frame selection unit checks a section in which the person and the object are interacting from the captured image, and uniformly selects the predetermined number of frames from the section in which the person and the object are interacting.

3. The model learning device according to claim 1 , further comprising a convergence condition determination unit that determines whether the updated parameters satisfy a convergence condition.

4. The model learning device according to claim 1 , wherein the model update unit calculates a loss using the replaced predetermined number of frames and the uniformly selected predetermined number of frames, and updates parameters of the model based on the loss.

5. 2. The model learning device according to claim 1, wherein the model updating unit extracts a first feature from the replaced predetermined number of frames based on a neural network model capable of image recognition that has been trained in advance, extracts a second feature from the uniformly selected predetermined number of frames based on the neural network model, calculates a third feature by applying a neural network composed of three fully connected layers to a first equation expressing a relationship between the first feature and the second feature using an attention mechanism with the first feature as a query, calculates a fourth feature by applying the neural network composed of three fully connected layers to a second equation expressing the relationship between the second feature and the first feature using the attention mechanism with the second feature as a query, calculates a loss using the third feature and the fourth feature, and updates a parameter of the model based on the loss.

6. A learning data acquisition unit that acquires learning data including a captured image of a person manipulating an object; an extraction unit that extracts support and queries to be used in learning based on the learning data; a frame selection unit that uniformly selects a predetermined number of frames from the captured images; a determination unit for determining whether the person and an object are interacting in the uniformly selected frames; a frame replacement unit that replaces the frame determined not to have had an interaction with a frame that is closest to or before or after the frame in which the person and the object have an interaction; a model updating unit that updates model parameters based on the replaced predetermined number of frames and the uniformly selected predetermined number of frames; A storage unit for storing the updated model; an action recognition unit that acquires inference data including an image of an interaction between the person and an object, and generates an action label using the acquired inference data and a model stored in the storage unit as input; an output control unit that controls the generated behavior label to be output to an output device; An action recognition device comprising:

7. A model learning method executed by a processor of a model learning device, acquiring learning data including captured images of a person manipulating an object; extracting support and queries for use in training based on the training data; uniformly selecting a predetermined number of frames from the captured images; determining whether the person and an object are interacting in the uniformly selected frames; replacing the frames determined not to be interacting with the human and the object based on a predetermined criterion with frames in which the human and the object are interacting; updating parameters of a model based on the replaced number of frames and the uniformly selected number of frames; The model learning method includes:

8. 1. A model learning program comprising instructions for execution by a processor of a model learning device, the instructions comprising: acquiring learning data including captured images of a person manipulating an object; extracting support and queries for use in training based on the training data; uniformly selecting a predetermined number of frames from the captured images; determining whether the person and an object are interacting in the uniformly selected frames; replacing the frames determined not to be interacting with the human and the object based on a predetermined criterion with frames in which the human and the object are interacting; updating parameters of a model based on the replaced number of frames and the uniformly selected number of frames; A model learning program that includes:

Images