Learning device for action recognition device, action recognition device, learning method, and learning program

The learning device improves behavior recognition by occluding irrelevant background areas in the learning process, enabling the model to learn state changes caused by actions, thus addressing shortcut learning issues and enhancing recognition accuracy.

WO2024241437A1PCT designated stage expired Publication Date: 2024-11-28NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/018985
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-22
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing behavior recognition models face challenges in improving recognition performance and generalization ability due to shortcut learning, which focuses on irrelevant video information, and simply increasing model complexity is insufficient for accurate action recognition.

Method used

The proposed solution involves a learning device with a first action estimation section, a second video generation section, and a parameter update section that occludes background areas unrelated to the action subject and object interactions, allowing the model to learn the characteristics of state changes caused by the action, thereby improving generalization and recognition accuracy.

Benefits of technology

This approach enhances the recognition performance and generalization ability of the model by focusing on relevant information, leading to more precise behavior recognition across various video types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023018985_28112024_PF_FP_ABST
    Figure JP2023018985_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The one aspect of the present invention provides a learning device for an action recognition device, the leaning device comprising a first action estimation unit, a second video generation unit, a second action estimation unit, and a parameter update unit. The first action estimation unit estimates, on the basis of a feature amount obtained from input first video containing time-series frame images and a parameter of a first action recognition model, first information relating to an action of an agent in the first video. The second video generation unit generates a second video in which a background region other than a region of the agent interacting with an object and a region of the object interacting with the agent in the first video is masked. The second action estimation unit estimates second information relating to the action of the agent in the second video on the basis of a feature amount obtained from the second video and a parameter of a second action recognition model. The parameter update unit updates the parameter of the first action recognition model on the basis of a difference between the first information and correct answer information relating to the action of the agent and a difference between the first information and the second information.
Need to check novelty before this filing date? Find Prior Art

Description

Learning device for behavior recognition device, behavior recognition device, learning method, and learning program

[0001] The embodiments relate to a learning device for an activity recognition device, an activity recognition device, a learning method, and a learning program.

[0002] Analyzing people's behavior is useful in a variety of situations, such as factories, retail stores, and event facilities. For example, analyzing the behavior of workers in factories, retail stores, etc. can help detect errors made by workers in their work and identify work-related risks. Analyzing worker behavior also makes work visible, which can be expected to result in improved work efficiency. In addition, analyzing the behavior of people visiting event facilities, for example, can help detect people behaving suspiciously. The applications of behavior recognition are endless.

[0003] One method for analyzing human behavior is to recognize the behavior of an actor, such as a person, present in a video. This type of analysis method is called behavior recognition. For example, Non-Patent Document 1 describes a method for recognizing the behavior of an actor from RGB information in a video.

[0004] Action recognition is formulated as a classification problem of estimating which of a predefined list of actions an actor's action corresponds to. In action recognition training, the relationship between video and action labels is typically learned end-to-end (End2End). However, End2End training can result in shortcut learning, which focuses on video information unrelated to the actor's action, resulting in insufficient recognition performance and generalization ability of the model. Therefore, as described in Non-Patent Document 2, for example, a method has been proposed for improving generalization ability by suppressing shortcut learning of the model.

[0005] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He. SlowFast Networks for Video Recognition. in Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV). pp. 6202-6211. 2019. Ying Zhang, Tao Xiang, Timothy M. Hospedales, Huchuan Lu. Deep Mutual Learning. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4320-4328. 2018.

[0006] Even if the method of suppressing shortcut learning of the model described in Non-Patent Document 2 is applied to the model learning process described in Non-Patent Document 1, it may not be possible to expect improvement in the recognition performance of the model or acquisition of generalization ability. This shows that simply increasing the number of models and implicitly learning the subject's actions using only pairs of video and action labels is insufficient for action recognition.

[0007] For example, when considering the action of "a person opening a refrigerator," one of the characteristics of this action is that "the person grabs the handle on the refrigerator door with their hand and pulls it, causing the refrigerator to change from a closed state to an open state." By suppressing learning of video information unrelated to the subject's action, while allowing the model to learn the characteristics of the object's state that change in response to the subject's action, further improvements in the model's recognition performance and generalization ability are expected.

[0008] The embodiments provide a learning device, an action recognition device, a learning method, and a learning program for an action recognition device that can improve the generalization ability of a model regardless of the type of video and recognize the action of an actor with higher accuracy.

[0009] A learning device for an action recognition device according to one aspect includes a first action estimation unit, a second image generation unit, a second action estimation unit, and a parameter update unit. The first action estimation unit estimates first information about an action of an actor in the first image based on feature amounts obtained from an input first image including a time-series of frame images and parameters of a first action recognition model. The second image generation unit generates a second image in which a background area other than an area of ​​the actor interacting with an object and an area of ​​the object interacting with the actor in the first image is masked. The second action estimation unit estimates second information about the action of the actor in the second image based on feature amounts obtained from the second image and parameters of the second action recognition model. The parameter update unit updates the parameters of the first action recognition model based on a difference between the first information and correct answer information about the action of the actor and a difference between the first information and the second information.

[0010] The embodiments can provide a learning device, an action recognition device, a learning method, and a learning program for an action recognition device that can improve the generalization ability of a model regardless of the type of video and recognize the action of an actor with higher accuracy.

[0011] FIG. 1 is a block diagram showing the configuration of the behavior recognition device according to the first embodiment. FIG. 2 is a flowchart showing the operation of the first behavior estimation unit according to the first embodiment. FIG. 3 is a flowchart showing the operation of the second image generation unit according to the first embodiment. FIG. 4 is a flowchart showing the operation of the second behavior estimation unit according to the first embodiment. FIG. 5 is a flowchart showing the operation of updating parameters of the first behavior recognition model by the parameter update unit according to the first embodiment. FIG. 6 is a flowchart showing the operation of updating parameters of the second behavior recognition model by the parameter update unit according to the first embodiment. FIG. 7 is a flowchart showing the operation during learning processing of the behavior recognition device according to the first embodiment. FIG. 8 is a flowchart showing the operation during inference processing of the behavior recognition device according to the first embodiment. FIG. 9 is a block diagram showing the configuration of the behavior recognition device according to the second embodiment. FIG. 10 is a flowchart showing the operation of the third image generation unit according to the second embodiment. FIG. 11 is a flowchart showing the operation of the fourth image generation unit according to the second embodiment. FIG. 12 is a flowchart showing the operation of the third behavior estimation unit according to the second embodiment. FIG. 13 is a flowchart showing the operation of the fourth behavior estimation unit according to the second embodiment. FIG. 14 is a flowchart showing the operation of updating parameters of the first behavior recognition model by the parameter update unit according to the second embodiment. Fig. 15 is a flowchart showing the operation of updating parameters of a third behavior recognition model by the parameter update unit in the second embodiment. Fig. 16 is a flowchart showing the operation of updating parameters of a fourth behavior recognition model by the parameter update unit in the second embodiment. Fig. 17 is a flowchart showing the operation during learning processing of the behavior recognition device in the second embodiment. Fig. 18 is a block diagram showing an example of the hardware configuration of the behavior recognition device according to each embodiment.

[0012] Hereinafter, embodiments will be described with reference to the drawings. (First Embodiment) The first embodiment will be described. FIG. 1 is a block diagram showing the configuration of a behavior recognition device according to the first embodiment. The behavior recognition device 1 is a computer that outputs an estimation result of the behavior of an actor in an input video. The actor is, for example, a person, but does not necessarily have to be a person. In the first embodiment, a neural network model, for example, is used to estimate the behavior. In the embodiment, not only information about the actor but also information about an object interacting with the actor is used to train the neural network model. The behavior recognition device 1 in the first embodiment includes a first behavior estimation unit 11, a second video generation unit 12, a second behavior estimation unit 13, a parameter update unit 14, and a parameter storage unit 15.

[0013] The first behavior estimation unit 11 estimates the behavior of the subject in the video input to the behavior recognition device 1 using the first behavior recognition model, and outputs an behavior label distribution representing the behavior estimation result. The behavior label distribution is a distribution of predicted probabilities for each behavior included in a predefined behavior list. Therefore, the number of dimensions of the behavior label distribution is equal to the number of types of behavior included in the predefined behavior list. Hereinafter, the behavior label distribution output from the first behavior estimation unit 11 will be referred to as a first behavior label distribution.

[0014] The video input to the behavior recognition device 1 is a time-series of frame images showing the behavior of the actor. The video is not particularly limited as long as it shows the behavior of the actor. For example, the video may be a first-person perspective video or a third-person perspective video. The first-person perspective video is video seen from the viewpoint of the actor. The first-person perspective video is obtained by capturing an image from a camera placed on the head or chest of the actor, for example, in accordance with the viewpoint direction of the actor. The third-person perspective video is video seen from a third-person perspective of the actor. The third-person perspective video is obtained by capturing an image of the actor from a camera placed above the actor, for example. In the following description, the video input to the behavior recognition device 1 is referred to as a first video.

[0015] The first action recognition model may be any model that converts features extracted from the first video into an action label distribution. For example, the first action recognition model may be configured as a single-label classification model or a multi-label classification model. If configured as a single-label classification model, the first action recognition model may be a neural network model including, for example, a linear layer and a softmax function as an activation function. If configured as a multi-label classification model, the first action recognition model may be a neural network model including, for example, a linear layer and a sigmoid function or a tanh function as an activation function.

[0016] The second image generator 12 generates a second image from the first image input to the behavior recognition device 1 and inputs the second image to the second behavior estimator 13. The second image is an image in which the area of ​​the first image, other than the area of ​​the body part of the actor interacting with the object and the area of ​​the object interacting with the actor, is occluded as a background area. For example, the second image of a first image of "a person opening a refrigerator" is an image in which the area other than the person's hands and the refrigerator is occluded. The second image generator 12 detects the area of ​​the body part of the actor interacting with the object and the area of ​​the object interacting with the actor in the first image using, for example, an object detector. The configuration of the object detector is not particularly limited. For example, a Faster-Region-based CNN (Faster-RCNN) can be used as the object detector. Furthermore, the method for generating an image in which the background area is occluded is also not limited. The occlusion of the background area can be achieved, for example, by blacking out the background area detected by the object detector, i.e., by setting all pixel values ​​of the background area to 0. Alternatively, the background region can be masked by smoothing the background region using, for example, a Gaussian filter.

[0017] The second behavior estimation unit 13 estimates the behavior of the subject in the second video using the second behavior recognition model and outputs an behavior label distribution representing the behavior estimation result. Hereinafter, the behavior label distribution output from the second behavior estimation unit 13 will be referred to as a second behavior label distribution.

[0018] The second action recognition model may be any model that converts features extracted from the second video into an action label distribution. For example, the second action recognition model may be configured as a single-label classification model or a multi-label classification model. If configured as a single-label classification model, the second action recognition model may be a neural network model including, for example, a linear layer and a softmax function as an activation function. If configured as a multi-label classification model, the second action recognition model may be a neural network model including, for example, a linear layer and a sigmoid function or a tanh function as an activation function. The second action recognition model may have the same configuration as the first action recognition model, or may have a different configuration.

[0019] The parameter update unit 14 updates the parameters of the first behavior recognition model and the second behavior recognition model stored in the parameter storage unit 15 in the learning process.

[0020] The parameter storage unit 15 stores parameters of neural network models as the first behavior recognition model and the second behavior recognition model. Furthermore, when the second video generation unit 12 generates the second video using a neural network model, the parameter storage unit 15 may store parameters of the neural network model for generating the second video.

[0021] Next, a description will be given of the operation of the behavior recognition device 1 in the first embodiment. Below, the operation of each element of the behavior recognition device 1 will be specifically described, with the operation divided into a learning process and an inference process.

[0022] <<<Learning Process>>> First, the learning process will be described. Fig. 2 is a flowchart showing the operation of the first behavior inferrer 11 in the first embodiment. First, the first behavior inferrer 11 acquires a first video (step S11). As described above, the first video is a time-series of frame images showing the behavior of the performer.

[0023] After acquiring the first video, the first behavior estimation unit 11 converts the first video into feature quantities (step S12). The feature quantities are quantities that represent the characteristics of the video expressed in vector form. The feature quantities are obtained, for example, by inputting the first video into a feature extraction network such as SlowFast and Video Swing Transformer.

[0024] After obtaining the feature amounts, the first behavior estimation unit 11 inputs the feature amounts into the first behavior recognition model to generate a first behavior label distribution (step S13).

[0025] After generating the first behavior label distribution, the first behavior estimation unit 11 outputs the first behavior label distribution to the parameter update unit 14 (step S14).

[0026] 3 is a flowchart showing the operation of the second video generator 12 in the first embodiment. First, the second video generator 12 acquires a first video (step S21).

[0027] Next, the second image generation unit 12 detects, in the first image, an area of ​​the body part of the actor that is interacting with the object and an area of ​​the object that is interacting with the actor using an object detector such as Faster-RCNN, and then generates a second image in which the background area other than the detected areas of the actor and object is masked (step S22).

[0028] Next, the second video generator 12 outputs the second video to the second behavior estimator 13 (step S23).

[0029] 4 is a flowchart showing the operation of the second behavior inferrer 13 in the first embodiment. First, the second behavior inferrer 13 acquires a second video (step S31).

[0030] After acquiring the second video, the second behavior inferrer 13 converts the second video into feature quantities (step S32). The feature quantities may be the same as those of the first behavior inferrer 11. Alternatively, the feature quantities may be different from those of the first behavior inferrer 11.

[0031] After obtaining the feature amounts, the second behavior estimation unit 13 inputs the feature amounts into the second behavior recognition model to generate a second behavior label distribution (step S33).

[0032] After generating the second behavior label distribution, the second behavior estimation unit 13 outputs the second behavior label distribution to the parameter update unit 14 (step S34).

[0033] 5 is a flowchart showing the operation of updating the parameters of the first behavior recognition model by the parameter update unit 14 in the first embodiment. First, the parameter update unit 14 acquires the first behavior label distribution, the second behavior label distribution, and the correct behavior label distribution (step S41). The correct behavior label distribution expresses the behavior of the subject appearing in the video as 1 and other behaviors as 0.

[0034] Next, the parameter update unit 14 calculates the difference (error) between the first behavior label distribution and the correct behavior label distribution (step S42). The expression of the difference between the first behavior label distribution and the correct behavior label distribution can be any expression of difference that can update the parameters of the first behavior recognition model. For example, the difference is calculated as the cross entropy of the first behavior label distribution with respect to the correct behavior label distribution. Specifically, the cross entropy of the first behavior label distribution with respect to the correct behavior label distribution is calculated by: 0 (x), the first behavior label distribution is y 1 When (x) is used, it is calculated from the following formula (1).

[0035] Next, the parameter update unit 14 calculates the difference between the first action label distribution and the second action label distribution (step S43). The expression of the difference between the first action label distribution and the second action label distribution may be any expression of difference that can update the parameters of the first action recognition model. For example, the difference is calculated as the Kullback-Leibler divergence of the second action label distribution with respect to the first action label distribution. Specifically, the Kullback-Leibler divergence of the second action label distribution with respect to the first action label distribution is calculated by: 1 (x), the second behavior label distribution is y 2 When (x) is used, it is calculated from the following formula (2).

[0036] Next, the parameter update unit 14 updates the parameters of the first behavior recognition model stored in the parameter storage unit 15 so as to minimize the difference calculated in steps S42 and S43 (step S44). The parameter update method is not particularly limited as long as it is a method that has the constraint of minimizing the difference calculated in steps S42 and S43.

[0037] 6 is a flowchart showing the operation of updating parameters of the second behavior recognition model by the parameter update unit 14 in the first embodiment. First, the parameter update unit 14 acquires the first behavior label distribution, the second behavior label distribution, and the correct behavior label distribution (step S51).

[0038] Next, the parameter update unit 14 calculates the difference between the second behavior label distribution and the correct behavior label distribution (step S52). The expression of the difference between the second behavior label distribution and the correct behavior label distribution can be any expression of difference that can update the parameters of the second behavior recognition model. For example, the difference is calculated as the cross entropy of the second behavior label distribution with respect to the correct behavior label distribution. Specifically, the cross entropy of the second behavior label distribution with respect to the correct behavior label distribution is calculated by dividing x by y, where x is an behavior label that represents a certain behavior that the subject can take, and y is the correct behavior label distribution. 0 (x), the second behavior label distribution is y 2 When (x) is used, it is calculated from the following formula (3).

[0039] Next, the parameter update unit 14 calculates the difference between the second action label distribution and the first action label distribution (step S53). The expression of the difference between the second action label distribution and the first action label distribution can be any expression of difference that can update the parameters of the second action recognition model. For example, the difference is calculated as the Kullback-Leibler divergence of the first action label distribution with respect to the second action label distribution. Specifically, the Kullback-Leibler divergence of the first action label distribution with respect to the second action label distribution is calculated by: 1 (x), the second behavior label distribution is y 2When (x) is used, it is calculated from the following formula (4).

[0040] Next, the parameter update unit 14 updates the parameters of the second behavior recognition model stored in the parameter storage unit 15 so as to minimize the difference calculated in steps S52 and S53 (step S54). The parameter update method is not particularly limited as long as it is a method that has the constraint of minimizing the difference calculated in steps S52 and S53.

[0041] 7 is a flowchart showing the operation of the learning process of the behavior recognition device 1 in the first embodiment. During the learning process, a large number of first videos are input to the behavior recognition device 1 as learning data. The behavior recognition device 1 repeats the processes of steps S61 to S64 shown in FIG. 7 for each piece of learning data until the parameters converge. The processes of steps S61 to S64 will be specifically described below.

[0042] First, the behavior recognition device 1 performs the operation of the first behavior estimator 11 described in Fig. 2 on the first video input as learning data. The behavior recognition device 1 also performs the operation of the second video generator 12 described in Fig. 3 on the first video input as learning data. The behavior recognition device 1 then performs the operation of the second behavior estimator 13 described in Fig. 4 on the second video generated by the operation of the second video generator 12 (step S61). Here, the operation of the first behavior estimator 11 and the operations of the second video generator 12 and the second behavior estimator 13 may be performed in parallel or sequentially.

[0043] Next, the behavior recognition device 1 performs an operation of updating the parameters of the first behavior recognition model as the operation of the parameter update unit 14 described in FIG. 5 using the first behavior label distribution, the second behavior label distribution, and the correct behavior label distribution (step S62).

[0044] After updating the parameters of the first behavior recognition model, the behavior recognition device 1 again executes the operation of the first behavior estimator 11 described in Fig. 2 on the first video. The behavior recognition device 1 also again executes the operation of the second video generator 12 described in Fig. 3 on the first video input as training data. Then, the behavior recognition device 1 again executes the operation of the second behavior estimator 13 described in Fig. 4 on the second video generated by the operation of the second video generator 12 (step S63). Here, the operation of the first behavior estimator 11 and the operations of the second video generator 12 and the second behavior estimator 13 may be performed in parallel or sequentially.

[0045] Next, the behavior recognition device 1 performs an operation of updating the parameters of the second behavior recognition model as the operation of the parameter update unit 14 described in FIG. 6 using the first behavior label distribution, the second behavior label distribution, and the correct behavior label distribution (step S64).

[0046] <<<<Inference Processing>>> Next, the inference processing will be described. Fig. 8 is a flowchart showing the operation of the behavior recognition device 1 during the inference processing in the first embodiment. During the inference processing, the behavior recognition device 1 uses the first behavior estimator 11 to infer the behavior of the performer in response to the input first video. That is, during the inference processing, the second video generation unit 12, the second behavior estimator 13, and the parameter updater 14 do not operate.

[0047] First, the first behavior inferrer 11 acquires a first video (step S71). As described above, the first video is a time-series of frame images showing the behavior of the subject. After acquiring the first video, the first behavior inferrer 11 converts the first video into feature quantities (step S72). After obtaining the feature quantities, the first behavior inferrer 11 inputs the feature quantities into a first behavior recognition model to generate a first behavior label distribution (step S73). The operations from step S71 to step S73 are the same as steps S11 to S13 in the learning process.

[0048] During the inference process, after generating the first activity label distribution, the first activity estimation unit 11 determines an activity label from the first activity label distribution (step S74). In the case of single-label classification, the first activity estimation unit 11 selects, for example, an activity label corresponding to an element in the activity label distribution that has the largest predicted probability value. In the case of multi-label classification, the first activity estimation unit 11 selects, for example, an activity label corresponding to an element in the activity label distribution that has a predicted probability value equal to or greater than a predefined label determination threshold.

[0049] After determining the activity label, the first activity estimation unit 11 outputs the determined activity label to, for example, a display device (step S75). As a result, the estimation result of the activity for the input first video can be displayed on the display device. Here, the output destination of the activity label is not limited to the display device.

[0050] Here, during the inference process in the first embodiment, the second image generation unit 12, the second behavior estimation unit 13, and the parameter update unit 14 are not operated. Therefore, if the behavior recognition device 1 only performs the inference process, the behavior recognition device 1 may have only the first behavior estimation unit 11 and the parameter storage unit 15 that stores parameters of the trained first behavior recognition model. The parameters of the first behavior recognition model are learned by a learning device that has the first behavior estimation unit 11, the second image generation unit 12, the second behavior estimation unit 13, the parameter update unit 14, and the parameter storage unit 15 shown in FIG. 1 . Furthermore, the parameter storage unit 15 may be provided separately from the behavior recognition device 1. In this case, the behavior recognition device 1 may acquire the parameters of the first behavior recognition model from the separately provided parameter storage unit 15 and estimate the behavior of the performer using the first behavior estimation unit 11.

[0051] As described above, in the first embodiment, the behavior recognition device 1 includes the first behavior estimation unit 11 that estimates the behavior of an actor using the entire first video and the second behavior estimation unit 13 that estimates the behavior of an actor using the second video in which the area of ​​the first video where the actor interacts with an object and the background area other than the area of ​​the object where the actor interacts with the actor are masked. The first behavior recognition model used in the first behavior estimation unit 11 is trained using the difference between the first behavior label distribution and the second behavior label distribution, in addition to the difference between the correct behavior label distribution and the first behavior label distribution. Conversely, the second behavior recognition model used in the second behavior estimation unit 13 is trained using the difference between the second behavior label distribution and the first behavior label distribution, in addition to the difference between the correct behavior label distribution and the second behavior label distribution. This allows the first behavior recognition model to learn features such as the state of an object that changes in response to the actor's behavior while suppressing learning of video information unrelated to the actor's behavior. Therefore, further improvement in the recognition performance and generalization ability of the first behavior recognition model is expected. For example, knowledge related to human behavior, such as the shape of a person's hands during an action or the open / close status of a refrigerator, can be embedded into a model that learns the relationship between video and action labels in an End2End manner, making it possible to focus on appropriate information about actions contained in the video and recognize human behavior more precisely.

[0052] Second Embodiment A second embodiment will be described. Fig. 9 is a block diagram showing the configuration of a behavior recognition device according to the second embodiment. Here, in Fig. 9, elements corresponding to those in Fig. 1 are assigned the same reference numerals as in Fig. 1. The behavior recognition device 1 in the second embodiment includes a first behavior estimation unit 11, a third image generation unit 121, a fourth image generation unit 122, a third behavior estimation unit 131, a fourth behavior estimation unit 132, a parameter update unit 14, and a parameter storage unit 15. That is, in the second embodiment, the second image generation unit 12 is divided into the third image generation unit 121 and the fourth image generation unit 122, and the second behavior estimation unit 13 is divided into the third behavior estimation unit 131 and the fourth behavior estimation unit 132.

[0053] Similar to the first behavior estimation unit 11 described in the first embodiment, the first behavior estimation unit 11 uses a first behavior recognition model to estimate the behavior of the subject in the first video input to the behavior recognition device 1, and outputs a behavior label distribution representing the behavior estimation result.

[0054] The third video generation unit 121 generates a third video from the first video input to the behavior recognition device 1, and inputs the third video to the third behavior estimation unit 131. The third video is a video in which the areas of the first video other than the areas of the body parts of the actor interacting with the object are occluded as background areas. For example, the third video of a first video of "a person opening a refrigerator" is a video in which the areas other than the person's hands are occluded.

[0055] The fourth image generator 122 generates a fourth image from the first image input to the behavior recognition device 1, and inputs the fourth image to the fourth behavior estimator 132. The fourth image is an image in which the area of ​​the first image other than the area of ​​the object that is interacting with the actor is occluded as a background area. For example, the fourth image of the first image of "a person opening a refrigerator" is an image in which the area other than the refrigerator is occluded.

[0056] The third image generation unit 121 detects, for example, an object detector to detect a region of the body part of the actor that interacts with the object in the first image. Similarly, the fourth image generation unit 122 detects, for example, an object detector to detect a region of the object that interacts with the actor in the first image. The configuration of the object detector is not particularly limited. For example, Faster-RCNN may be used as the object detector. Furthermore, the method for generating an image in which the background region is occluded is also not limited. The occlusion of the background region may be achieved, for example, by painting the background region detected by the object detector black, i.e., by setting all pixel values ​​of the background region to 0. Alternatively, the occlusion of the background region may be achieved, for example, by smoothing the background region using a Gaussian filter.

[0057] The third behavior estimation unit 131 estimates the behavior of the subject in the third video using the third behavior recognition model and outputs an behavior label distribution representing the behavior estimation result. The fourth behavior estimation unit 132 estimates the behavior of the subject in the fourth video using the fourth behavior recognition model and outputs an behavior label distribution representing the behavior estimation result. Hereinafter, the behavior label distribution output from the third behavior estimation unit 131 will be referred to as the third behavior label distribution. Also, the behavior label distribution output from the fourth behavior estimation unit 132 will be referred to as the fourth behavior label distribution.

[0058] The third behavior recognition model may be any model that converts feature quantities extracted from the third video into an behavior label distribution. Similarly, the fourth behavior recognition model may be any model that converts feature quantities extracted from the fourth video into an behavior label distribution. For example, the third behavior recognition model and the fourth behavior recognition model may be configured as single-label classification models or multi-label classification models. When configured as single-label classification models, the third behavior recognition model and the fourth behavior recognition model may be neural network models including, for example, a linear layer and a softmax function as an activation function. When configured as multi-label classification models, the third behavior recognition model and the fourth behavior recognition model may be neural network models including, for example, a linear layer and a sigmoid function or a tanh function as an activation function. The third behavior recognition model and the fourth behavior recognition model may have the same configuration as the first behavior recognition model or may have a different configuration.

[0059] The parameter update unit 14 updates the parameters of the first behavior recognition model, the third behavior recognition model, and the fourth behavior recognition model stored in the parameter storage unit 15 in the learning process.

[0060] The parameter storage unit 15 stores parameters of neural network models as the first behavior recognition model, the third behavior recognition model, and the fourth behavior recognition model. Furthermore, when the generation of the third video by the third video generation unit 121 and the generation of the fourth video by the fourth video generation unit 122 are performed using neural network models, the parameter storage unit 15 may store parameters of the neural network models for generating the third video and the fourth video.

[0061] Next, a description will be given of the operation of the behavior recognition device 1 in the second embodiment. Below, the operation of each element of the behavior recognition device 1 will be specifically described, with the operation divided into a learning process and an inference process.

[0062] <<<Learning Process>>> First, the learning process will be described. Here, the operation of the first behavior inferrer 11 in the second embodiment is the same as the operation in FIG. 2 described in the first embodiment. Therefore, the description will be omitted.

[0063] 10 is a flowchart showing the operation of the third video generation unit 121 in the second embodiment. First, the third video generation unit 121 acquires the first video (step S81).

[0064] Next, the third image generation unit 121 detects the area of ​​the body part of the actor that is interacting with the object in the first image using an object detector such as Faster-RCNN, and then generates a third image in which the background area other than the detected area of ​​the body part of the actor is masked (step S82).

[0065] Next, the third video generator 121 outputs the third video to the third behavior estimator 131 (step S83).

[0066] 11 is a flowchart showing the operation of the fourth video image generator 122 in the second embodiment. First, the fourth video image generator 122 acquires the first video image (step S91).

[0067] Next, the fourth image generation unit 122 detects an object area that interacts with the actor in the first image using an object detector such as Faster-RCNN, and then generates a fourth image in which the background area other than the detected object area is masked (step S92).

[0068] Next, the fourth video generator 122 outputs the fourth video to the fourth behavior estimator 132 (step S93).

[0069] 12 is a flowchart showing the operation of the third behavior inferrer 131 in the second embodiment. First, the third behavior inferrer 131 acquires a third video (step S101).

[0070] After acquiring the third video, the third behavior inferrer 131 converts the third video into feature quantities (step S102). The feature quantities may be the same as those of the first behavior inferrer 11. Alternatively, the feature quantities may be different from those of the first behavior inferrer 11.

[0071] After obtaining the feature amounts, the third behavior estimation unit 131 inputs the feature amounts into a third behavior recognition model to generate a third behavior label distribution (step S103).

[0072] After generating the third behavior label distribution, the third behavior estimation unit 131 outputs the third behavior label distribution to the parameter update unit 14 (step S104).

[0073] 13 is a flowchart showing the operation of the fourth behavior inferrer 132 in the second embodiment. First, the fourth behavior inferrer 132 acquires a fourth video (step S111).

[0074] After acquiring the fourth video, the fourth behavior inferrer 132 converts the fourth video into feature quantities (step S112). The feature quantities may be the same as those of the first behavior inferrer 11. Alternatively, the feature quantities may be different from those of the first behavior inferrer 11.

[0075] After obtaining the feature amounts, the fourth behavior estimation unit 132 inputs the feature amounts into a fourth behavior recognition model to generate a fourth behavior label distribution (step S113).

[0076] After generating the fourth behavior label distribution, the fourth behavior estimation unit 132 outputs the fourth behavior label distribution to the parameter update unit 14 (step S114).

[0077] 14 is a flowchart showing the operation of updating parameters of the first behavior recognition model by the parameter update unit 14 in the second embodiment. First, the parameter update unit 14 acquires the first behavior label distribution, the third behavior label distribution, the fourth behavior label distribution, and the correct behavior label distribution (step S121).

[0078] Next, the parameter update unit 14 calculates the difference between the first behavior label distribution and the correct behavior label distribution (step S122). The expression of the difference between the first behavior label distribution and the correct behavior label distribution may be any expression of difference capable of updating the parameters of the first behavior recognition model. For example, the difference is calculated as the cross entropy of the first behavior label distribution with respect to the correct behavior label distribution. The cross entropy is calculated using the above-mentioned formula (1).

[0079] Next, the parameter update unit 14 calculates the difference between the first action label distribution and the third and fourth action label distributions (step S123). The expression of the difference between the first action label distribution and the third and fourth action label distributions may be any expression of difference that can update the parameters of the first action recognition model. For example, the difference is calculated as the average sum of the Kullback-Leibler divergence of the third action label distribution with respect to the first action label distribution and the Kullback-Leibler divergence of the fourth action label distribution with respect to the first action label distribution. Specifically, the average sum is calculated by dividing the action label representing a certain action that the subject can take by x and the first action label distribution by y. 1 (x), the third behavior label distribution is y 3 (x), the fourth behavior label distribution is y 4 When (x) is used, it is calculated from the following formula (5).

[0080] Alternatively, the difference may be calculated as the Kullback-Leibler divergence between the first behavior label distribution and the distribution obtained by combining the third behavior label distribution and the fourth behavior label distribution. Specifically, the difference may be calculated using the following formula (6):

[0081] Next, the parameter update unit 14 updates the parameters of the first behavior recognition model stored in the parameter storage unit 15 so as to minimize the difference calculated in steps S122 and S123 (step S124). The parameter update method is not particularly limited as long as it is a method that has the constraint of minimizing the difference calculated in steps S122 and S123.

[0082] 15 is a flowchart showing the operation of updating parameters of the third behavior recognition model by the parameter update unit 14 in the second embodiment. First, the parameter update unit 14 acquires the first behavior label distribution, the third behavior label distribution, the fourth behavior label distribution, and the correct behavior label distribution (step S131).

[0083] Next, the parameter update unit 14 calculates the difference between the third behavior label distribution and the correct behavior label distribution (step S132). The expression of the difference between the third behavior label distribution and the correct behavior label distribution can be any expression of difference that can update the parameters of the third behavior recognition model. For example, the difference is calculated as the cross entropy of the third behavior label distribution with respect to the correct behavior label distribution. Specifically, the cross entropy of the third behavior label distribution with respect to the correct behavior label distribution is calculated by: 0 (x), the third behavior label distribution is y 3 When (x) is used, it is calculated from the following formula (7).

[0084] Next, the parameter update unit 14 calculates the difference between the third action label distribution and the first and fourth action label distributions (step S133). The expression of the difference between the third action label distribution and the first and fourth action label distributions may be any expression of difference that can update the parameters of the third action recognition model. For example, the difference is calculated as the average sum of the Kullback-Leibler divergence of the first action label distribution with respect to the third action label distribution and the Kullback-Leibler divergence of the fourth action label distribution with respect to the third action label distribution. Specifically, the average sum is calculated by dividing the action label representing a certain action that the subject can take by x and the first action label distribution by y. 1 (x), the third behavior label distribution is y 3 (x), the fourth behavior label distribution is y 4 When (x) is used, it is calculated from the following formula (8).

[0085] Furthermore, the difference may be calculated as the Kullback-Leibler divergence between the third behavior label distribution and a distribution obtained by combining the first behavior label distribution and the fourth behavior label distribution. Specifically, the difference may be calculated using the following formula (9):

[0086] Next, the parameter update unit 14 updates the parameters of the third behavior recognition model stored in the parameter storage unit 15 so as to minimize the difference calculated in steps S132 and S133 (step S134). The parameter update method is not particularly limited as long as it is a method that has the constraint of minimizing the difference calculated in steps S132 and S133.

[0087] 16 is a flowchart showing the operation of updating parameters of the fourth behavior recognition model by the parameter update unit 14 in the second embodiment. First, the parameter update unit 14 acquires the first behavior label distribution, the third behavior label distribution, the fourth behavior label distribution, and the correct behavior label distribution (step S141).

[0088] Next, the parameter update unit 14 calculates the difference between the fourth behavior label distribution and the correct behavior label distribution (step S142). The expression of the difference between the fourth behavior label distribution and the correct behavior label distribution can be any expression of difference that can update the parameters of the fourth behavior recognition model. For example, the difference is calculated as the cross entropy of the fourth behavior label distribution with respect to the correct behavior label distribution. Specifically, the cross entropy of the fourth behavior label distribution with respect to the correct behavior label distribution is calculated by: 0 (x), the fourth behavior label distribution is y 4 When (x) is used, it is calculated from the following formula (10).

[0089] Next, the parameter update unit 14 calculates the difference between the fourth action label distribution and the first and third action label distributions (step S143). The expression of the difference between the fourth action label distribution and the first and third action label distributions may be any expression of difference that can update the parameters of the fourth action recognition model. For example, the difference is calculated as the average sum of the Kullback-Leibler divergence of the first action label distribution with respect to the fourth action label distribution and the Kullback-Leibler divergence of the third action label distribution with respect to the fourth action label distribution. Specifically, the average sum is calculated by denoting an action label representing a certain action that the subject can take as x and the first action label distribution as y. 1 (x), the third behavior label distribution is y 3 (x), the fourth behavior label distribution is y 4 When (x) is used, it is calculated from the following formula (11).

[0090] The difference may be calculated as the Kullback-Leibler divergence between the fourth behavior label distribution and a distribution obtained by combining the first behavior label distribution and the third behavior label distribution. Specifically, the difference may be calculated using the following formula (12):

[0091] Next, the parameter update unit 14 updates the parameters of the fourth behavior recognition model stored in the parameter storage unit 15 so as to minimize the difference calculated in steps S142 and S143 (step S144). The parameter update method is not particularly limited as long as it is a method that has the constraint of minimizing the difference calculated in steps S142 and S143.

[0092] Fig. 17 is a flowchart showing the operation of the learning process of the behavior recognition device 1 in the second embodiment. During the learning process, a large number of first videos are input to the behavior recognition device 1 as learning data. The behavior recognition device 1 repeats the processes of steps S151 to S156 shown in Fig. 17 for each piece of learning data until the parameters converge. The processes of steps S151 to S156 will be specifically described below.

[0093] First, the behavior recognition device 1 performs the operation of the first behavior estimator 11 described in FIG. 2 on the first video input as training data. The behavior recognition device 1 also performs the operation of the third video generator 121 described in FIG. 10 on the first video input as training data. The behavior recognition device 1 then performs the operation of the third behavior estimator 131 described in FIG. 12 on the third video generated by the operation of the third video generator 121. The behavior recognition device 1 also performs the operation of the fourth video generator 122 described in FIG. 11 on the first video input as training data. The behavior recognition device 1 then performs the operation of the fourth behavior estimator 132 described in FIG. 13 on the fourth video generated by the operation of the fourth video generator 122 (step S151). Here, the operation of the first behavior estimator 11, the operations of the third video generator 121 and the third behavior estimator 131, and the operations of the fourth video generator 122 and the fourth behavior estimator 132 may be performed in parallel or sequentially.

[0094] Next, the behavior recognition device 1 performs the parameter update operation of the first behavior recognition model as the operation of the parameter update unit 14 described in FIG. 14 using the first behavior label distribution, the third behavior label distribution, the fourth behavior label distribution, and the correct behavior label distribution (step S152).

[0095] After updating the parameters of the first behavior recognition model, the behavior recognition device 1 again executes the operation of the first behavior estimator 11 described in Fig. 2 on the first video. Furthermore, the behavior recognition device 1 again executes the operation of the third video generator 121 described in Fig. 10 on the first video input as training data. Then, the behavior recognition device 1 again executes the operation of the third behavior estimator 131 described in Fig. 12 on the third video generated by the operation of the third video generator 121. Furthermore, the behavior recognition device 1 again executes the operation of the fourth video generator 122 described in Fig. 11 on the first video input as training data. Then, the behavior recognition device 1 again executes the operation of the fourth behavior estimator 132 described in Fig. 13 on the fourth video generated by the operation of the fourth video generator 122 (step S153). Here, the operation of the first behavior estimation unit 11, the operation of the third image generation unit 121 and the third behavior estimation unit 131, and the operation of the fourth image generation unit 122 and the fourth behavior estimation unit 132 may be performed in parallel or sequentially.

[0096] Next, the behavior recognition device 1 performs the parameter update operation of the third behavior recognition model as the operation of the parameter update unit 14 described in FIG. 15 using the first behavior label distribution, the third behavior label distribution, the fourth behavior label distribution, and the correct behavior label distribution (step S154).

[0097] After updating the parameters of the third behavior recognition model, the behavior recognition device 1 again executes the operation of the first behavior estimator 11 described in Fig. 2 on the first video. Furthermore, the behavior recognition device 1 again executes the operation of the third video generator 121 described in Fig. 10 on the first video input as training data. Then, the behavior recognition device 1 again executes the operation of the third behavior estimator 131 described in Fig. 12 on the third video generated by the operation of the third video generator 121. Furthermore, the behavior recognition device 1 again executes the operation of the fourth video generator 122 described in Fig. 11 on the first video input as training data. Then, the behavior recognition device 1 again executes the operation of the fourth behavior estimator 132 described in Fig. 13 on the fourth video generated by the operation of the fourth video generator 122 (step S155). Here, the operation of the first behavior estimation unit 11, the operation of the third image generation unit 121 and the third behavior estimation unit 131, and the operation of the fourth image generation unit 122 and the fourth behavior estimation unit 132 may be performed in parallel or sequentially.

[0098] Next, the behavior recognition device 1 performs the parameter update operation of the fourth behavior recognition model as the operation of the parameter update unit 14 described in FIG. 16 using the first behavior label distribution, the third behavior label distribution, the fourth behavior label distribution, and the correct behavior label distribution (step S156).

[0099] <<<Inference Processing>>> Next, the inference processing will be described. In the second embodiment, the inference processing is performed according to Fig. 8 as in the first embodiment. That is, in the second embodiment as well, during the inference processing, the behavior recognition device 1 uses the first behavior estimation unit 11 to estimate the behavior of the performer in response to the input first video. During the inference processing, the third video generation unit 121, the fourth video generation unit 122, the third behavior estimation unit 131, the fourth behavior estimation unit 132, and the parameter update unit 14 do not operate.

[0100] During the inference process in the second embodiment, the third image generation unit 121, the fourth image generation unit 122, the third behavior estimation unit 131, the fourth behavior estimation unit 132, and the parameter update unit 14 are not operated. Therefore, if the behavior recognition device 1 performs only the inference process, the behavior recognition device 1 may have only the first behavior estimation unit 11 and the parameter storage unit 15 that stores parameters of the trained first behavior recognition model. The parameters of the first behavior recognition model are learned by a learning device that has the first behavior estimation unit 11, the third image generation unit 121, the fourth image generation unit 122, the third behavior estimation unit 131, the fourth behavior estimation unit 132, the parameter update unit 14, and the parameter storage unit 15. In addition, the parameter storage unit 15 may be provided separately from the behavior recognition device 1. In this case, the behavior recognition device 1 may acquire parameters of the first behavior recognition model from the separately provided parameter storage unit 15 and estimate the behavior of the operator using the first behavior estimation unit 11.

[0101] As described above, in the second embodiment, the behavior recognition device 1 includes the first behavior estimation unit 11 that estimates the behavior of an actor using the entire first video, the third behavior estimation unit 131 that estimates the behavior of an actor using a third video in which a background area other than the area of ​​the actor interacting with an object in the first video is masked, and the fourth behavior estimation unit 132 that estimates the behavior of an actor using a fourth video in which a background area other than the area of ​​the object interacting with the actor in the first video is masked. The first behavior recognition model used by the first behavior estimation unit 11 is trained using the difference between the correct behavior label distribution and the first behavior label distribution, as well as the difference between the first behavior label distribution and the third behavior label distribution and the difference between the first behavior label distribution and the fourth behavior label distribution. Meanwhile, the third behavior recognition model used by the third behavior estimation unit 131 is trained using the difference between the correct behavior label distribution and the third behavior label distribution, as well as the difference between the third behavior label distribution and the first behavior label distribution and the difference between the third behavior label distribution and the fourth behavior label distribution. Furthermore, in training the fourth behavior recognition model used in the fourth behavior estimation unit 132, in addition to the difference between the correct behavior label distribution and the fourth behavior label distribution, the difference between the fourth behavior label distribution and the first behavior label distribution and the difference between the fourth behavior label distribution and the third behavior label distribution are used. As a result, similar to the first embodiment, in the second embodiment, the first behavior recognition model can learn features such as the state of an object that has changed in response to the behavior of the performer while suppressing learning of video information unrelated to the behavior of the performer. Therefore, further improvement in the recognition performance and generalization ability of the first behavior recognition model is expected.

[0102] Furthermore, in the second embodiment, only the area of ​​the actor needs to be detected to generate the third video, and only the area of ​​the object needs to be detected to generate the fourth video. Therefore, the parameters of the object detector used to generate the third video are optimized to detect the area of ​​the actor in the first video. Similarly, the parameters of the object detector used to generate the fourth video are optimized to detect the area of ​​the object in the first video. Therefore, it is expected that the detection performance of each object detector will be improved. This is also expected to improve the accuracy of the behavior estimation of the third behavior estimation unit 131 and the fourth behavior estimation unit 132.

[0103] Here, in the second embodiment, the third behavior estimation unit 131 generates a third behavior label distribution for a third video in which an area other than the area of ​​the actor interacting with the object is occluded, and the fourth behavior estimation unit 132 generates a fourth behavior label distribution for a fourth video in which an area other than the area of ​​the object interacting with the actor is occluded. That is, behavior label distributions are estimated separately for the third video and the fourth video. The behavior recognition device 1 of the second embodiment may have, instead of the third behavior estimation unit 131 and the fourth behavior estimation unit 132, a single behavior estimation unit that receives the third video and the fourth video as input and outputs an behavior label distribution for the combination of the third video and the fourth video.

[0104] Next, a description will be given of an example hardware configuration of the behavior recognition device 1. Fig. 18 is a block diagram showing an example hardware configuration of the behavior recognition device 1 according to each embodiment. For example, the behavior recognition device 1 may be a computer including a processor 101, a program memory 102, a data memory 103, an input / output interface 104, and a communication interface 105. The processor 101, the program memory 102, the data memory 103, the input / output interface 104, and the communication interface 105 are connected via a bus 106 so as to be able to communicate with each other.

[0105] The processor 101 is a processor equipped with hardware that controls the behavior recognition device 1. The processor 101 is, for example, a CPU (Central Processing Unit). The processor 101 may be a MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or the like instead of a CPU. The processor 101 does not need to be configured by a single CPU or the like, but may be configured by multiple CPUs or the like. The processor 101 can operate as the first behavior estimation unit 11, the second image generation unit 12, the second behavior estimation unit 13, and the parameter update unit 14 in the first embodiment described above by operating in accordance with a program stored in the program memory 102. The processor 101 can also operate as the first behavior estimation unit 11, the third image generation unit 121, the fourth image generation unit 122, the third behavior estimation unit 131, the fourth behavior estimation unit 132, and the parameter update unit 14 in the second embodiment described above by operating in accordance with a program stored in the program memory 102. Note that the operations of the first behavior estimation unit 11, the second image generation unit 12, the second behavior estimation unit 13, the third image generation unit 121, the fourth image generation unit 122, the third behavior estimation unit 131, the fourth behavior estimation unit 132, and the parameter update unit 14 may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) and an FPGA (Field-Programmable Gate Array).

[0106] The program memory 102 is a storage medium that stores a program executed by the processor 101. The program memory 102 may be a storage medium such as a hard disk drive (HDD) or a solid state drive (SSD).

[0107] The data memory 103 is a storage medium for storing various types of data. The data memory includes a read-only memory (ROM), a random access memory (RAM), and a storage. The ROM stores various setting values ​​and the like required for the operation of the behavior recognition device 1. The RAM is used as a working memory during processing by the processor 101. The storage may store parameters of each behavior recognition model described above as the parameter storage unit 15. The storage area serving as the parameter storage unit 15 may be provided in an external storage medium such as a universal serial bus (USB) memory, or in a storage device such as a database server located in the cloud.

[0108] An input device 200 and an output device 300 used by a user or the like are connected to the input / output interface 104. The input device 200 is a device that accepts user operations, such as a keyboard, touch panel, touchpad, or mouse. The user's operation of the input device 200 is accepted by the behavior recognition device 1 via the input / output interface 104. The output device 300 is one of various output devices, such as a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, or an audio output device such as a speaker. The output device 300 performs output based on data sent from the input / output interface 104. The input device 200 and the output device 300 may be built into the behavior recognition device 1. Alternatively, the input device 200 and the output device 300 may be input devices and output devices of another information terminal that can communicate with the behavior recognition device 1 via a network NW.

[0109] The communication interface 105 includes, for example, one or more wireless or wired communication interface units, and enables transmission and reception of information to and from the communication network NW. For example, a wireless LAN (Local Area Network) can be used as the wireless interface, and a wired LAN can be used as the wired interface.

[0110] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0111] REFERENCE SIGNS LIST 1... Behavior recognition device 11... First behavior estimation unit 12... Second image generation unit 13... Second behavior estimation unit 14... Parameter update unit 15... Parameter storage unit 121... Third image generation unit 122... Fourth image generation unit 131... Third behavior estimation unit 132... Fourth behavior estimation unit 101... Processor 102... Program memory 103... Data memory 104... Input / output interface 105... Communication interface 106... Bus 200... Input device 300... Output device

Claims

1. A learning device for a behavior recognition device comprising: a first behavior estimation unit that estimates first information regarding an actor's behavior in a first video based on features obtained from an input first video including time-series frame images and parameters of a first behavior recognition model; a second image generation unit that generates a second video in which a background area other than an area of ​​the actor interacting with an object and an area of ​​the object interacting with the actor in the first video is occluded; a second behavior estimation unit that estimates second information regarding the actor's behavior in the second video based on features obtained from the second video and parameters of a second behavior recognition model; and a parameter update unit that updates parameters of the first behavior recognition model based on a difference between the first information and correct answer information regarding the actor's behavior and a difference between the first information and the second information.

2. A learning device for an action recognition device as described in claim 1, wherein the parameter update unit further updates parameters of the second action recognition model based on a difference between the second information and the correct answer information and a difference between the first information and the second information.

3. A first behavior estimation unit that estimates first information regarding an actor's behavior in the first video based on a feature amount obtained from an input first video including time-series frame images and parameters of a first behavior recognition model; a third image generation unit that generates a third video in which a background area in the first video is occluded other than an area of ​​the actor having an interaction with an object; a fourth image generation unit that generates a fourth video in which a background area in the first video is occluded other than an area of ​​an object having an interaction with the actor; a third behavior estimation unit that estimates third information regarding the actor's behavior in the third video based on the feature amount obtained from the third video and parameters of the third behavior recognition model; a fourth behavior estimation unit that estimates fourth information regarding the actor's behavior in the fourth video based on the feature amount obtained from the fourth video and parameters of the fourth behavior recognition model; a parameter update unit that updates parameters of the first behavior recognition model based on a difference between the first information and correct answer information regarding the actor's behavior and differences between the first information and the third information and fourth information; A learning device for an activity recognition device comprising:

4. The learning device for the behavior recognition device described in claim 3, wherein the parameter update unit further updates parameters of the third behavior recognition model based on a difference between the third information and the correct answer information and differences between the third information and the first information and fourth information, and updates parameters of the fourth behavior recognition model based on a difference between the fourth information and the correct answer information and differences between the fourth information and the first information and third information.

5. A behavior recognition device comprising a first behavior estimation unit that estimates first information regarding the behavior of an actor in the first video based on the first video and parameters of the first behavior recognition model updated by a learning device for a behavior recognition device described in any one of claims 1 to 4.

6. A learning method comprising: estimating first information regarding an actor's behavior in a first video based on features obtained from an input first video including time-series frame images and parameters of a first behavior recognition model; generating a second video in which background areas other than an area of ​​the actor interacting with an object and an area of ​​the object interacting with the actor in the first video are occluded; estimating second information regarding the actor's behavior in the second video based on features obtained from the second video and parameters of a second behavior recognition model; and updating the parameters of the first behavior recognition model based on a difference between the first information and correct answer information regarding the actor's behavior and a difference between the first information and the second information.

7. A program for causing a processor to function as each of the components of a learning device for an action recognition device according to any one of claims 1 to 4.