Behavior recognition device, behavior recognition method, and behavior recognition program

Abstract

To enhance classification accuracy between classes having similar behavior features.SOLUTION: According to an embodiment, a behavior recognition device includes: a data acquisition part for acquiring input data including a first moving image segment; a potential feature vector generation part for generating a potential feature vector by inputting a moving image segment in a potential feature generation model including a pre-learned behavior recognition model and an encoder; a behavior label generation part for generating a behavior label by inputting the potential feature vector in a pre-learned behavior classification model; and an output control part for outputting the behavior label.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an action recognition device, an action recognition method, and an action recognition program. [Background technology]

[0002] Currently, due to a labor shortage, digitalization of operations is required in various industries. When digitalizing operations, it is necessary to understand how the operations to be digitized are being carried out. However, accurately understanding the operations is difficult. For example, even if an operations manual is prepared to understand the operations, there are cases where the manual and the actual operations are different, making it difficult to accurately understand the operations.

[0003] There are methods for visualizing human behavior using a camera, etc. For example, Non-Patent Document 1 discloses a technology for visualizing a person's actual behavior using behavior recognition technology in order to automatically acquire human behavior from a camera.

[0004] Furthermore, Non-Patent Document 2 discloses a technique that is capable of learning a certain class of behavior from multiple video images. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He, “SlowFast Networks for Video Recognition”, ICCV2019. [Non-patent document 2] Jiahao Wang, Yunhong Wang, Sheng Liu, Annan Li, “Few-shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning”, ACMMM2021. [Non-patent document 3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” NAACL, 2019. [Non-patent document 4] Kenichiro Ishii, Shuko Ueda, Eisaku Maeda, Hiroshi Murase, "Easy-to-understand Pattern Recognition", Ohmsha. Summary of the Invention [Problem to be solved by the invention]

[0006] Non-Patent Document 1 has a problem in that it is costly to define the actions to be recognized (definition of action labels).

[0007] Furthermore, the technique disclosed in Non-Patent Document 2 has a problem in that it is difficult to classify classes having similar behavioral features.

[0008] The object of the present invention has been made in light of the above circumstances, and its purpose is to provide a technology that can improve the stability and accuracy of learning by generating features that take into account behavioral and linguistic features and that can classify classes with similar behavioral features, and by applying criteria using the generated features to a method for selecting support data in episodes. [Means for solving the problem]

[0009] The behavior recognition device according to the embodiment includes a data acquisition unit that acquires input data including a first video segment, a latent feature vector generation unit that generates a latent feature vector by inputting the video segment into a latent feature generation model that includes a pre-trained behavior recognition model and an encoder, a behavior label generation unit that generates a behavior label by inputting the latent feature vector into a pre-trained behavior classification model, and an output control unit that outputs the behavior label. [Effects of the Invention]

[0010] According to one aspect of the present invention, By generating features that take into account behavioral and linguistic features that can classify classes with similar behavioral features, and applying criteria using the generated features to select support data for episodes, the stability of learning can be improved, leading to improved accuracy. A behavior recognition device, a behavior recognition method, and a behavior recognition program can be provided. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an example of a hardware configuration of a behavior recognition apparatus according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the software configuration of the behavior recognition device according to the embodiment in association with the hardware configuration shown in FIG. [Figure 3] FIG. 3 is a flowchart showing an example of a learning procedure of the behavior recognition device according to this embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a conceptual diagram of a neural network model of the latent feature generation model construction unit. [Figure 5] FIG. 5 is a flowchart showing an example of detailed processing for generating an episode set in step ST103. [Figure 6] FIG. 6 is a flowchart showing an example of detailed processing for updating the behavior classification model in step ST104. [Figure 7] FIG. 7 is a flowchart showing an example of an activity label estimation procedure of the activity recognition device according to this embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a conceptual diagram of a neural network model of the behavior classification unit. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, a behavior recognition device, a behavior recognition method, and a behavior recognition program will be described in detail with reference to the drawings. In the following embodiments, parts with the same numbers perform similar operations, and redundant explanations will be omitted. For example, when there are multiple identical or similar elements, a common symbol may be used to describe each element without distinguishing between them, or a subnumber may be used in addition to the common symbol to describe each element with distinction between them.

[0013] [Embodiment] First, in the following description, an "action" is an action that represents a human movement such as walking, running, or carrying, and an "action label" is a verb that represents an action such as walking or running.

[0014] Non-Patent Document 2 uses a learning method called "few-shot learning." Few-shot learning constructs training data by generating a large amount of task data called episodes. Specifically, suppose there is training data with a large number of classes, from which five classes are extracted. If each class holds three samples as training data, the "support data" becomes a 5-way 3-shot (5 classes, 3 samples) classification task. In this case, if the "query data" holds one sample each of test data for two of the five classes, the task of few-shot learning is to learn using the support data and accurately classify each sample of the two classes. Here, this combination of support data and query data is called an "episode."

[0015] However, the few-shot learning used in Non-Patent Document 2 generates a large number of episodes and further trains a model that has been trained in advance from a large amount of training data. However, the episode construction method is random sampling, and if the class selection method is incorrect initially, the neural network will fall into a local solution, affecting generalization performance and reducing accuracy. In particular, there is a problem of accuracy being affected when classes with low separability, which is the ratio of inter-class variance to intra-class variance, are mixed. Specifically, the technology disclosed in Non-Patent Document 2 has difficulty classifying classes with similar behavioral features.

[0016] In the following description, as in Non-Patent Document 2, an "episode" refers to a combination of the above-mentioned support data and query data.

[0017] Furthermore, in the following description, a "video segment" refers to a unit consisting of a predetermined number of video frames. For example, 30 frames may be considered as one segment. For example, when considering a long-term video sequence of hundreds or thousands of frames, a video segment may be a collection of frames sampled at regular intervals.

[0018] (composition) FIG. 1 is a block diagram showing an example of a hardware configuration of a behavior recognition device 1 according to an embodiment. The behavior recognition device 1 is a computer that analyzes input data, generates output data, and outputs the output data. The behavior recognition device 1 may be installed in any location.

[0019] 1, the behavior recognition device 1 includes a control unit 10, a program storage unit 20, a data storage unit 30, a communication interface 40, and an input / output interface 50. The control unit 10, the program storage unit 20, the data storage unit 30, the communication interface 40, and the input / output interface 50 are communicatively connected to one another via a bus. Furthermore, the communication interface 40 may be communicatively connected to an external device via a network. Furthermore, the input / output interface 50 is communicatively connected to an input device 51 and an output device 52.

[0020] The control unit 10 controls the behavior recognition device 1. The control unit 10 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 10 may be an integrated circuit capable of executing various programs.

[0021] The program storage unit 20 may use, as a storage medium, a combination of nonvolatile memory that can be written to and read from at any time, such as an EPROM (Erasable Programmable Read Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive), and a nonvolatile memory such as a ROM (Read Only Memory). The program storage unit 20 stores programs necessary for executing various processes. That is, the control unit 10 can realize various controls and operations by reading and executing the programs stored in the program storage unit 20.

[0022] The data storage unit 30 is a storage that uses a combination of nonvolatile memory such as a HDD or memory card, which can be written to and read from at any time, and volatile memory such as RAM (Random Access Memory), as a storage medium. The data storage unit 30 is used to store data acquired and generated in the process of the control unit 10 executing programs and performing various processes.

[0023] The communication interface 40 includes one or more wired or wireless communication modules. For example, the communication interface 40 includes a communication module for wired or wireless connection to an external device via a network. The communication interface 40 may also include a wireless communication module for wireless connection to an external device such as a Wi-Fi access point or base station. Furthermore, the communication interface 40 may also include a wireless communication module for wireless connection to an external device using short-range wireless technology. In other words, the communication interface 40 may be any general communication interface as long as it can communicate with an external device under the control of the control unit 10 and send and receive various information including past performance data.

[0024] The input / output interface 50 is connected to an input device 51, an output device 52, etc. The input / output interface 50 is an interface that enables transmission and reception of information between the input device 51 and the output device 52. The input / output interface 50 may be integrated with the communication interface 40. For example, the behavior recognition device 1 and at least one of the input device 51 and the output device 52 may be wirelessly connected using short-range wireless technology or the like, and information may be transmitted and received using the short-range wireless technology.

[0025] The input device 51 includes, for example, a keyboard, a pointing device, etc., which are used by the user to input various data to the behavior recognition device 1. The input device 51 may also include a reader for reading data to be stored in the program storage unit 20 or the data storage unit 30 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0026] The output device 52 includes a display or the like that displays images to be displayed from the behavior recognition device 1 to the administrator of the behavior recognition device 1.

[0027] FIG. 2 is a block diagram showing the software configuration of the behavior recognition device 1 according to the embodiment in association with the hardware configuration shown in FIG. The control unit 10 includes a data acquisition unit 101, a latent feature generation model construction unit 102, a learning data generation unit 103, a model update unit 104, a behavior classification unit 105, and an output control unit 106.

[0028] The data acquisition unit 101 acquires training data during training. For example, the training data may be a collection of video segments, action labels, and object labels. Here, the action labels are verbs that express actions such as walking and running, as described above. The object labels may be general labels used in object detection, etc. Furthermore, the data acquisition unit 101 acquires input data during behavior inference. The input data may include video segments.

[0029] The latent feature generation model construction unit 102 constructs a latent feature generation model. The latent feature generation model construction unit 102 constructs a latent feature generation model that receives video segments included in the received training data as input and outputs a latent feature vector. The latent feature generation model construction unit 102 may output parameters of the constructed latent model to the training data generation unit 103. Alternatively, the latent feature generation model construction unit 102 may output information for using the latent feature generation model in the training data generation unit 103 to the training data generation unit 103.

[0030] The training data generation unit 103 includes a set extraction unit 1031 , a latent feature vector generation unit 1032 , a separability calculation unit 1033 , an episode set addition unit 1034 , a determination unit 1035 , and a memory control unit 1036 .

[0031] The learning data generation unit 103 generates a set of episodes. The learning data generation unit 103 receives as input video segments, parameters of a latent feature generation model, and a set of action labels, and outputs a set of episodes to be used for learning.

[0032] The set extraction unit 1031 randomly extracts a set of N-way K-shot support data and query data from a set of video segments and action labels.

[0033] The latent feature vector generation unit 1032 generates a latent feature vector. The latent feature vector generation unit 1032 constructs a latent feature generation model from the parameters received from the latent feature generation model construction unit 102, and generates a latent feature vector for each video segment of the support data using the constructed latent feature generation model.

[0034] The separability calculation unit 1033 calculates the separability using the latent feature vector. Note that a specific method for calculating the separability will be described later.

[0035] The episode set adding unit 1034 adds the degree of separation and the episode (combination of support data and query data) as an episode set.

[0036] The determination unit 1035 determines whether the number of episode sets is greater than a predetermined number.

[0037] The storage control unit 1036 stores the episode sets in the learning data storage unit 301. The storage control unit 1036 stores in the learning data storage unit 301 episode sets that have exceeded a predetermined number.

[0038] The model update unit 104 includes a batch extraction unit 1041 , a loss calculation unit 1042 , a model parameter update unit 1043 , a separability threshold update unit 1044 , a convergence determination unit 1045 , and a memory control unit 1046 .

[0039] The model update unit 104 updates the behavior classification model. During learning, the model update unit 104 acquires parameters of the behavior classification model used by the behavior classification unit 105, and acquires data necessary for learning the behavior classification model. Here, the behavior classification model may be a general model as long as it can classify behavior labels. Then, the model update unit 104 updates the model parameters based on the loss calculated from the acquired data, and stores the updated parameters in the model storage unit 302.

[0040] The batch extraction unit 1041 acquires a set of episodes from the training data storage unit 301. From the acquired set of episodes, the batch extraction unit 1041 extracts a batch of episodes whose degree of separation is equal to or greater than a threshold. Here, a batch is a training unit for a neural network, and is a group of multiple episodes.

[0041] The loss calculation unit 1042 estimates the behavior label of the batch and calculates the loss. Here, the method of estimating the behavior label of the batch and the method of calculating the loss will be described in detail later.

[0042] The model parameter update unit 1043 updates the model parameters by backpropagating the loss.

[0043] The separability threshold updating unit 1044 updates the separability threshold. The method for updating the separability threshold will be described in detail later.

[0044] The convergence determination unit 1045 determines whether the loss value is less than a threshold value.

[0045] The storage control unit 1046 stores the updated parameters in the model storage unit 302. The storage control unit 1046 may store the updated parameters in the model storage unit 302 when the loss value becomes less than the threshold value.

[0046] The behavior classification unit 105 includes a latent feature vector generation unit 1051 and a behavior label generation unit 1052.

[0047] The latent feature vector generation unit 1051 generates a latent feature vector using a latent feature generation model. The latent feature generation model may be a model constructed by the latent feature generation model construction unit 102.

[0048] The behavior label generation unit 1052 generates behavior labels. The behavior label generation unit 1052 acquires parameters of the behavior classification model stored in the model storage unit 302 and generates behavior labels by inputting a latent feature vector into the behavior classification model. The behavior label generation unit 1052 outputs the generated behavior labels to the output control unit 106.

[0049] The output control unit 106 outputs the behavior label. The output control unit 106 outputs the behavior label to the output device 52 through the input / output interface 50. The output control unit 106 may then perform control so that the behavior label is displayed on the display of the output device 52.

[0050] (Learning behavior) First, the learning operation will be described. In this learning operation, it is assumed that the parameters of the behavior categorization model used by the behavior categorization unit 105 are stored in the model storage unit 302.

[0051] FIG. 3 is a flowchart showing an example of a learning procedure of the behavior recognition device 1 in this embodiment. The operation of this flowchart is realized by the control unit 10 of the behavior recognition device 1 reading and executing a program stored in the program storage unit 20. This operation starts when a user of the behavior recognition device 1 inputs learning data into the input device 51. Here, the learning data may be a collection of video segments, behavior labels, and object labels.

[0052] In step ST101, the data acquiring unit 101 acquires learning data. The input device 51 receives the learning data and outputs the learning data to the data acquiring unit 101. The data acquiring unit 101 receives the learning data and outputs the learning data to the latent feature generation model building unit 102.

[0053] In step ST102, the latent feature generation model construction unit 102 constructs a latent feature generation model. The latent feature generation model construction unit 102 constructs a latent feature generation model that receives as input video segments included in the received training data and outputs latent feature vectors. The latent feature generation model construction unit 102 may output parameters of the constructed latent model to the training data generation unit 103. Alternatively, the latent feature generation model construction unit 102 may output information for using the latent feature generation model in the training data generation unit 103 to the training data generation unit 103.

[0054] FIG. 4 is a diagram showing an example of a conceptual diagram of the neural network model of the latent feature generation model construction unit 102. The neural network model may be any neural network model as long as it can define input and output and is effective for the output target, such as a convolutional neural network or a neural network model using a Transformer. For example, in the example of Figure 2, the latent feature generation model includes a pre-trained activity recognition model 1021, an encoder 1022, a decoder 1023, and a language model 1024.

[0055] The pre-trained behavior recognition model 1021 may be any behavior recognition model such as the SlowFast model disclosed in Non-Patent Document 1.

[0056] The encoder 1022 and decoder 1023 may be a general encoder 1022 and decoder 1023, and a detailed description thereof will be omitted here.

[0057] The language model 1024 may be a pre-trained language model 1024 such as BERT as disclosed in Non-Patent Document 3.

[0058] 4 may be included in the training data and may be annotated to video segments in advance. If there is no annotation of the object label, the latent feature generation model construction unit 102 may extract the object label using a technique such as object detection.

[0059] 4, when a video segment is input, the pre-trained action recognition model 1021 outputs an action feature vector from the fully connected layer of the model. The output action feature vector is output to the encoder 1022.

[0060] The encoder 1022 generates a latent feature vector by encoding the behavioral feature vector. The generated latent feature vector may be, for example, a 256-dimensional vector. The generated latent feature vector is output to the decoder 1023.

[0061] The decoder 1023 reconstructs an action feature vector by decoding the latent feature vector, and also reconstructs a language feature vector obtained from the language model 1024. Here, the language feature vector is a language feature vector obtained from a combination of an action label and an interacting object label, etc.

[0062] When training the encoder 1022 and the decoder 1023 to reconstruct an action feature vector and a language feature vector from a latent feature vector, the latent feature generation model construction unit 102 calculates an L2 loss (first loss), which is the mean square error between the action feature vector obtained from the pre-trained action recognition model 1021 and the action feature vector generated by the decoder 1023 from the latent feature vector, and calculates an L2 loss (second loss) between the language feature vector output by the language model 1024 in response to input action labels and object labels and the language feature vector generated by the decoder 1023 from the latent feature vector. The latent feature generation model construction unit 102 then trains the encoder 1022 and the decoder 1023 to minimize these L2 losses, thereby enabling encoding of a desired latent feature vector. The loss may be an L1 loss, which is the mean absolute error, or the like.

[0063] By reconstructing the action feature vector in this way and simultaneously generating a language feature vector, it is possible to obtain a latent feature vector that takes into account the differences in the language feature vectors. As a result, it is possible to obtain a latent feature vector with high discriminatory ability for actions that are similar but have different meanings. For example, a typical action recognition model is not good at distinguishing between the actions of "push" and "pull." However, the language feature vectors for "push" and "pull" are different. Therefore, even if the action feature vectors are similar, the language feature vectors are different, so different latent feature vectors can be obtained for "push" and "pull."

[0064] Here, a trained latent feature generation model is sufficient to obtain a latent feature vector. Therefore, once the encoder 1022 is trained to obtain a latent feature vector, the decoder 1023 is not used to generate the latent feature vector unless new training is performed. That is, in the example of Figure 2, the latent feature generation model refers to the pre-trained activity recognition model 1021 and the encoder 1022.

[0065] That is, the latent feature vector is a vector obtained by using the pre-trained behavior recognition model 1021 as input an action feature vector that can be extracted from a video segment, and training the latent feature generation model (encoder 1022) to reconstruct a language feature vector extracted from the extracted action feature vector, the action label corresponding to the video segment, and the label of the interacting object.

[0066] Furthermore, when reconstructing the action feature vector and the language feature vector, commonly used losses such as the L1 norm or L2 norm of the original action feature vector and the reconstructed action feature vector, and the L1 norm or L2 norm of the original language feature vector and the reconstructed language feature vector, can be used as the reconstruction loss, thereby enabling the reconstruction of the action feature vector and the language feature vector. When actually performing reconstruction, the sum of the reconstruction loss of the action feature vector and the reconstruction loss of the language feature vector may be used as the overall loss. For example, reconstruction may be performed using a weighted sum of the reconstruction loss of the action feature vector and the reconstruction loss of the language feature vector.

[0067] 3, in step ST103, the learning data generation unit 103 generates a set of episodes. The learning data generation unit 103 receives as input video segments, parameters of a latent feature generation model, and a set of action labels, and outputs a set of episodes to be used for learning.

[0068] FIG. 5 is a flowchart showing an example of detailed processing for generating an episode set in step ST103. In step ST201, the set extraction unit 1031 randomly extracts sets of N-way K-shot support data and query data from a set of video segments and action labels. The set extraction unit 1031 of the training data generation unit 103 extracts sets of N-way K-shot support data and query data from the set of video segments and action labels among the input data. That is, the set extraction unit 1031 extracts sets of support data for N classes and K samples and query data for evaluation of the same classes as the extracted N classes. The number of combinations of support data and query data and the number of query data samples to prepare for each of the N classes may be determined experimentally.

[0069] In step ST202, the latent feature vector generation unit 1032 generates a latent feature vector. The latent feature vector generation unit 1032 constructs a latent feature generation model from the parameters received from the latent feature generation model construction unit 102, and generates a latent feature vector for each video segment of the support data using the constructed latent feature generation model. Alternatively, the latent feature vector generation unit 1032 generates a latent feature vector for each video segment of the support data using the latent feature generation model of the latent feature generation model construction unit 102.

[0070] In step ST203, the separability calculation unit 1033 calculates the separability using the latent feature vector. The separability calculation unit 1033 calculates the intra-class variance and inter-class variance of the support data based on the latent feature vector. Then, the separability calculation unit 1033 calculates the separability, which is the ratio of the inter-class variance to the intra-class variance.

[0071] Here, the intra-class variance is σ w 2 , the inter-class variance is σ b 2 and class C i The sample set belonging to X i Here, i is a positive integer greater than or equal to 1. Also, class c i The number of samples belonging to iLet the sample mean vector be m i Let the total number of samples be N i , where m is the mean vector of all samples, the intra-class variance and inter-class variance are expressed as follows:

[0072]

number

[0073] From the above formula, the degree of separation is σ b 2 / σ w 2 It is expressed as:

[0074] In step ST204, the episode set adding unit 1034 adds the degree of separation and the episode (combination of support data and query data) as an episode set.

[0075] In step ST205, the decision unit 1035 decides whether or not the number of episode sets is greater than a predetermined number. If the number of episode sets is equal to or less than the predetermined number, the process returns to step ST201. On the other hand, if the number of episode sets is greater than the predetermined number, the process proceeds to step ST206.

[0076] In step ST206, the storage control unit 1036 stores the set of episodes in the learning data storage unit 301.

[0077] 3 , in step ST104, the model update unit 104 updates the behavior categorization model. The model update unit 104 acquires parameters of the behavior categorization model used by the behavior categorization unit 105 during learning, and acquires data necessary for learning the behavior categorization model. Then, the model update unit 104 updates the model parameters based on the loss calculated from the acquired data, and stores the updated parameters in the model storage unit 302.

[0078] FIG. 6 is a flowchart showing an example of detailed processing for updating the behavior classification model in step ST104. In step ST301, the batch extraction unit 1041 acquires an episode set from the training data storage unit 301.

[0079] In step ST302, the batch extraction unit 1041 extracts a batch of episodes whose degree of separation is equal to or greater than a threshold value from the acquired set of episodes. Here, a batch is a learning unit of a neural network, and is a plurality of episodes.

[0080] In step ST303, the loss calculation unit 1042 estimates the behavior label of the batch and calculates the loss. The loss calculation unit 1042 estimates the behavior label of the batch. Specifically, the loss calculation unit 1042 inputs support data in multiple episodes in the batch into an behavior classification model and obtains the behavior label from the result of estimation by the behavior classification model. Then, the loss calculation unit 1042 calculates the loss based on the estimated behavior label and pre-defined behavior labels included in the training data. Here, the loss may be, for example, a cross-entropy loss that calculates the classification loss between the estimated behavior label and the pre-defined behavior label.

[0081] In step ST304, the model parameter update unit 1043 updates the model parameters by backpropagating the loss. The model parameter update unit 1043 acquires parameters used in the behavior classification model stored in the model storage unit 302. Then, the model parameter update unit 1043 updates the parameters by backpropagating the loss using a general method such as gradient descent.

[0082] In step ST305, the separability threshold update unit 1044 updates the separability threshold. For example, the model update unit 104 extracts a batch and uses it for learning if the separability (the ratio of inter-class variance to intra-class variance) exceeds the threshold up to a certain number of episodes. On the other hand, if the separability does not exceed the threshold, the batch is not extracted and therefore not used for learning. This prevents early learning data from being forgotten. Here, forgetting early learning data means that the parameters of the model are updated as the data learned in the early stages of learning learns different data, and the model adapts to the different data, resulting in a loss of separation performance for the early learning data. Furthermore, the separability threshold update unit 1044 updates the separability threshold to a smaller threshold for episodes that exceed a certain number. In this way, by using episodes with low separability as targets for random sampling in learning, early forgetting can be prevented while preventing a decrease in sampling patterns.

[0083] In step ST306, convergence determination unit 1045 determines whether the loss value is less than the threshold value. If it is determined that the loss value is less than the threshold value, that is, that sufficient learning has been achieved, convergence determination unit 1045 considers that the loss value has converged. Then, the processing proceeds to step ST307. On the other hand, if the loss value is equal to or greater than the threshold value, convergence determination unit 1045 determines that the loss value has not converged. Then, the processing returns to step ST302.

[0084] In step ST307, the storage control section 1046 stores the updated parameters in the model storage section 302.

[0085] In this way, the behavior recognition device 1 can update the parameters of the behavior categorization model through learning.

[0086] (inference behavior) Next, the processing performed by the control unit 10 during behavior inference will be described. FIG. 7 is a flowchart showing an example of an activity label estimation procedure of the activity recognition device 1 in this embodiment.

[0087] The operation of this flowchart is realized by the control unit 10 of the behavior recognition device 1 reading and executing a program stored in the program storage unit 20. This operation starts when a user of the behavior recognition device 1 inputs input data into the input device 51. Here, the input data may include a video segment.

[0088] In step ST401, the data acquiring unit 101 acquires input data. The input device 51 that has received the input data outputs the input data to the data acquiring unit 101. The data acquiring unit 101 that has received the input data outputs the input data to the latent feature generation model construction unit 102, the behavior recognition unit. Furthermore, the data acquiring unit 101 may output a signal indicating that the input data has been received to the latent feature generation model construction unit 102. In response to receiving the signal, the latent feature generation model construction unit 102 outputs parameters to be used in the latent feature generation model or information for using the latent feature generation model to the behavior classification unit 105.

[0089] In step ST402, the latent feature vector generation unit 1051 generates a latent feature vector using a latent feature generation model. Upon receiving input data from the data acquisition unit 101 and parameters to be used in the latent feature generation model from the latent feature generation model construction unit 102, the latent feature vector generation unit 1051 constructs a latent feature generation model and inputs video segments included in the input data to the latent feature generation model to generate a latent feature vector. Alternatively, the latent feature vector generation unit 1051 generates a latent feature vector using the latent feature generation model of the latent feature generation model construction unit 102.

[0090] In step ST403, the behavior label generation unit 1052 generates behavior labels. The behavior label generation unit 1052 acquires parameters of the behavior classification model stored in the model storage unit 302 and inputs a latent feature vector to the behavior classification model to generate behavior labels. The behavior label generation unit 1052 outputs the generated behavior labels to the output control unit 106.

[0091] FIG. 8 is a diagram showing an example of a conceptual diagram of the neural network model of the behavior classification unit 105. As shown in FIG. 8 includes a pre-trained activity recognition model 1021, an encoder 1022, and an activity classification model 1053. The pre-trained activity recognition model 1021 and the encoder 1022 configure a latent feature generation model. That is, the pre-trained activity recognition model 1021 and the encoder 1022 are used to generate the latent feature vector described in step ST401. Then, the activity classification model 1053 is used to generate the activity label described in step ST402.

[0092] In step ST404, the output control unit 106 outputs the behavior label. The output control unit 106 outputs the behavior label to the output device 52 through the input / output interface 50. Then, the output control unit 106 may perform control so that the behavior label is displayed on the display of the output device 52.

[0093] In this way, by inputting a video segment into the behavior recognition device 1, a behavior label corresponding to the video segment is generated, and the generated behavior label is displayed on the display of the output device 52.

[0094] (Effects of the embodiment) According to the embodiment described above, the behavior recognition device 1 can generate behavior labels by learning in consideration of the behavioral features and linguistic features of the behavior labels. Furthermore, the behavior recognition device 1 can also classify behavior labels with similar behavioral features.

[0095] [Other embodiments] It should be noted that the present invention is not limited to the above embodiment. For example, in the learning operation, learning of the encoder 1022 in step ST102 does not necessarily have to be executed. For example, once learning of the encoder 1022 has been completed, it may be executed as needed.

[0096] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0097] 1…Action recognition device 10...Control unit 101...Data acquisition section 102...Latent feature generation model construction unit 1021…Pre-trained behavior recognition model 1022...Encoder 1023...Decoder 1024…Language Model 103...Learning data generation unit 1031...Set extraction section 1032...Latent feature vector generation unit 1033…Separation degree calculation unit 1034…Episode collection addition section 1035…Judgment section 1036...Memory control unit 104...Model Update Section 1041...Batch extraction unit 1042…Loss calculation section 1043...Model parameter update unit 1044...Separation threshold update unit 1045...Convergence judgment unit 1046...Memory control unit 105…Behavior classification section 1051...Latent feature vector generation unit 1052...Action label generation unit 1053...Behavioral classification model 106...Output control unit 20...Program memory section 30...Data storage unit 301...Learning data storage unit 302...Model memory section 40...Communication interface 50...Input / output interface 51...Input device 52...Output device

Claims

1. a data acquisition unit that acquires input data including a first video segment, and acquires training data including a second video segment and an action label and an object label annotated to the second video segment when training an action classification model; a latent feature vector generation unit that generates a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model, an encoder, a decoder, and a language model; a latent feature generation model construction unit that calculates a first loss between an action feature vector obtained from the pre-trained action recognition model and an action feature vector generated by inputting the latent feature vector to the decoder, calculates a second loss between a first language feature vector generated by inputting the action label and the object label to the language model and a second language feature vector generated by inputting the latent feature vector to the decoder, and trains the encoder and the decoder so as to minimize the first loss and the second loss; an action label generation unit that generates an action label by inputting the latent feature vector into the action classification model; an output control unit that outputs the behavior label; An activity recognition device comprising:

2. a data acquisition unit that acquires input data including a first video segment, and acquires training data including a second video segment and an action label and an object label annotated to the second video segment when training an action classification model; a latent feature vector generation unit that generates a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model and an encoder; a model storage unit that stores parameters used by the behavior classification model; a training data generation unit that generates a set of episodes based on the video segments and the training data; a model update unit that acquires the parameters stored in the model storage unit, extracts batches from the set of episodes that satisfy a predetermined condition, estimates an action label of the extracted batch, calculates a loss between the estimated action label and an action label included in the training data, and updates the parameters by backpropagating the calculated loss; and an action label generation unit that generates an action label by inputting the latent feature vector into the action classification model; an output control unit that outputs the behavior label; The learning data generation unit comprises: a set extraction unit that extracts episodes, which are sets of N-way K-shot support data and query data, from the set of the second video segments and the action labels included in the training data; a latent feature vector generation unit that generates a second latent feature vector by inputting the second video segment into the latent feature generation model; a separability calculation unit that calculates a separability using the second latent feature vector; an episode set addition unit that adds the degree of separation and the episode to the episode set; Equipped with The model update unit extracts the batch and uses it for learning if the degree of separation exceeds a threshold up to a specific number of episodes, does not extract the batch and does not use it for learning if the degree of separation does not exceed the threshold, and updates the threshold to be smaller for episodes that exceed the specific number of episodes.

3. 3. The behavior recognition device according to claim 2, wherein the separability calculation unit calculates an inter-class variance and an intra-class variance of the support data based on the second latent feature vector, and calculates a separability that is a ratio of the inter-class variance to the intra-class variance.

4. acquiring input data including a first video segment, and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an activity classification model; generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model, an encoder, a decoder, and a language model; calculating a first loss between an action feature vector obtained from the pre-trained action recognition model and an action feature vector generated by inputting the latent feature vector to the decoder, calculating a second loss between a first language feature vector generated by inputting the action label and the object label to the language model and a second language feature vector generated by inputting the latent feature vector to the decoder, and training the encoder and the decoder to minimize the first loss and the second loss; generating an action label by inputting the latent feature vector into the action classification model; outputting the activity label; The activity recognition method includes:

5. acquiring input data including a first video segment, and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an action classification model; generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model and an encoder; storing parameters used by the behavior classification model; generating a set of episodes based on the video segments and the training data; acquiring the parameters, extracting batches from the set of episodes that satisfy a predetermined condition, estimating action labels of the extracted batches, calculating a loss between the estimated action labels and action labels included in the training data, and updating the parameters by backpropagating the calculated loss; generating an action label by inputting the latent feature vector into the action classification model; outputting the activity label; and generating the set of episodes comprises: extracting episodes, which are sets of N-way K-shot support data and query data, from the set of the second video segments and the action labels included in the training data; generating a second latent feature vector by inputting the second video segment into the latent feature generation model; calculating a degree of separation using the second latent feature vector; the degree of separation and the episodes as the episode set; Equipped with The behavior recognition method includes: updating the parameters by extracting the batch and using it for learning if the degree of separation exceeds a threshold up to a specific number of episodes; not extracting the batch and not using it for learning if the degree of separation does not exceed the threshold; and updating the threshold to be smaller for episodes that exceed the specific number of episodes.

6. acquiring input data including a first video segment, and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an activity classification model; generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model, an encoder, a decoder, and a language model; calculating a first loss between an action feature vector obtained from the pre-trained action recognition model and an action feature vector generated by inputting the latent feature vector to the decoder, calculating a second loss between a first language feature vector generated by inputting the action label and the object label to the language model and a second language feature vector generated by inputting the latent feature vector to the decoder, and training the encoder and the decoder to minimize the first loss and the second loss; generating an action label by inputting the latent feature vector into the action classification model; outputting the activity label; An action recognition program comprising instructions for causing a processor of an action recognition device to execute the above.

7. acquiring input data including a first video segment, and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an activity classification model; generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model and an encoder; storing parameters used by the behavior classification model; generating a set of episodes based on the video segments and the training data; acquiring the parameters, extracting batches from the set of episodes that satisfy a predetermined condition, estimating action labels of the extracted batches, calculating a loss between the estimated action labels and action labels included in the training data, and updating the parameters by backpropagating the calculated loss; generating an action label by inputting the latent feature vector into a pre-trained action classification model; outputting the activity label; and generating the episode set comprises: extracting episodes, which are sets of N-way K-shot support data and query data, from the set of the second video segments and the action labels included in the training data; generating a second latent feature vector by inputting the second video segment into the latent feature generation model; calculating a degree of separation using the second latent feature vector; the degree of separation and the episodes as the episode set; Equipped with The behavior recognition program includes updating the parameters by extracting the batch and using it for learning if the degree of separation exceeds a threshold up to a specific number of episodes, not extracting the batch and not using it for learning if the threshold is not exceeded, and updating the threshold to be smaller for episodes that exceed the specific number.

Images