Behavior recognition system, behavior recognition method, and program

Abstract

To provide an action recognition technology which can estimate actions in various scenes, and has robustness.SOLUTION: A pre-processing unit 2 converts an input voice into a mel-spectrogram. A first feature extraction unit 3 accepts input of the generated mel-spectrogram into a learned model to calculate a feature vector. A second feature extraction unit 4 accepts input of time-series RGB information composing an input video into the learned model to calculate the feature vector. A skeleton information extraction unit 5 extracts skeleton information in a time series from the input video. A third feature extraction unit 6 accepts input of the extracted time-series skeletal information into the learned model to calculate the feature vector. A combining unit 7 combines three calculated feature vectors to generate one feature vector. An action estimation unit 8 accepts input of the generated feature vector into the learned model to calculate a score for each action class.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an activity recognition technology. [Background technology]

[0002] Conventionally, behavior recognition technology is known as a technology for estimating the behavior of a person captured in an image. Among these behavior recognition technologies, 3D convolution-based (e.g., see Non-Patent Documents 1 and 2) and skeleton-based (e.g., see Non-Patent Documents 3 and 4) are known as methods for estimating behavior using deep learning technology with an image as input.

[0003] Of these two methods, the 3D convolution-based method obtains information useful for behavior estimation from video. Here, video is three-dimensional data that has a time axis in addition to two-dimensional spatial information (vertical and horizontal). From this three-dimensional data, video features are extracted using 3D convolution processing, taking into account changes in movement over time. Then, a person's behavior is estimated based on the extracted features.

[0004] On the other hand, skeleton-based methods first estimate the positions of major parts of a person's body, such as the head, hands, and waist, from an image to obtain position information for the entire human skeleton. Next, the skeleton position information is expressed as a graph structure, and features are extracted by performing graph convolution processing on multiple graph information with different time series. This allows features that take into account changes in a person's posture over time to be extracted from the video. Finally, the person's behavior is estimated based on the extracted features. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Christoph Feichtenhofer, et al. “SlowFast Networks for Video Recognition”, [online], [Retrieved December 14, 2020], Internet<https: / / arxiv.org / pdf / 1812.03982.pdf> [Non-Patent Document 2] Joao Carreira, et al. “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset”, [online], [searched on December 14, 2020], Internet<https: / / arxiv.org / pdf / 1705.07750.pdf> [Non-Patent Document 3] OpenPose, [online], [searched December 14, 2020], Internet<URL:https: / / github.com / CMU-Perceptual-Computing-Lab / openpose> [Non-Patent Document 4] Sijie Yan, et al. “Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition”, [online], [Retrieved December 14, 2020], Internet<https: / / arxiv.org / pdf / 1801.07455.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] The above two methods have different technical approaches, and therefore are effective in different situations for action estimation. Specifically, while the 3D convolution-based method can identify differences in actions due to the interaction with surrounding objects, it has difficulty in identifying subtle differences in actions. On the other hand, the skeleton-based method, in contrast to the 3D convolution-based method, can identify subtle differences in actions, but cannot identify differences in actions due to the interaction with surrounding objects. In addition, since the above two methods do not use audio as input, they are unable to identify actions where audio is important.

[0007] The present invention has been made in consideration of the above circumstances, and has an object to provide a robust behavior recognition technology capable of estimating behavior in a variety of situations. [Means for solving the problem]

[0009] A behavior recognition system according to a second embodiment of the present invention includes a pre-processing unit that converts the voice data of a pair of voice data and first video data into image data, a first feature extraction unit that inputs the generated image data into a first trained model and calculates a first feature vector, a skeletal information extraction unit that extracts skeletal information in time series from the first video data, a second feature extraction unit that inputs the extracted time-series skeletal information into a second trained model and calculates a second feature vector, a combination unit that combines the calculated first feature vector and second feature vector to generate a third feature vector, a video restoration unit that inputs the generated third feature vector into a third trained model and restores the second video data as an attention map, and a behavior estimation unit that inputs time-series RGB information constituting the first video data into a fourth trained model and calculates a score for each behavior class, the behavior estimation unit multiplying the value of each channel to be input into the fourth trained model by the attention map.

[0010] A behavior recognition system according to a third embodiment of the present invention includes a pre-processing unit that converts audio data, out of a pair of audio data and video data, into image data; a first feature extraction unit that inputs the generated image data into a first trained model and calculates a first feature vector; a second feature extraction unit that inputs time-series RGB information constituting the video data into a second trained model and calculates a second feature vector; a combination unit that combines the calculated first feature vector and second feature vector to generate a third feature vector; a third feature extraction unit that inputs the generated third feature vector into a third trained model and generates a fourth feature vector as an attention map; a skeletal information extraction unit that extracts skeletal information in time series from the video data; and a behavior estimation unit that inputs the extracted time-series skeletal information into a fourth trained model and calculates a score for each behavior class, the behavior estimation unit multiplying the value of each node of the skeleton to be input to the fourth trained model by the attention map.

[0012] A behavior recognition method according to a second embodiment of the present invention includes a step of converting the voice data of a pair of voice data and first video data into image data, a step of inputting the generated image data into a first trained model to calculate a first feature vector, a step of extracting skeletal information in time series from the first video data, a step of inputting the extracted time-series skeletal information into a second trained model to calculate a second feature vector, a step of combining the calculated first feature vector and second feature vector to generate a third feature vector, a step of inputting the generated third feature vector into a third trained model to restore the second video data as an attention map, and a step of inputting time-series RGB information constituting the first video data into a fourth trained model to calculate a score for each behavior class, wherein the value of each channel input into the fourth trained model is multiplied by the attention map.

[0013] A behavior recognition method according to a third embodiment of the present invention includes the steps of converting audio data of a pair of audio data and video data into image data, inputting the generated image data into a first trained model to calculate a first feature vector, inputting time-series RGB information constituting the video data into a second trained model to calculate a second feature vector, combining the calculated first feature vector and second feature vector to generate a third feature vector, inputting the generated third feature vector into a third trained model to generate a fourth feature vector as an attention map, extracting skeletal information in time series from the video data, inputting the extracted time-series skeletal information into a fourth trained model, and calculating a score for each behavior class, wherein the step is to multiply the value of each node of the skeleton to be input into the fourth trained model by the attention map.

[0015] A program according to a second embodiment of the present invention causes a computer to execute the following steps: converting the audio data of a pair of audio data and first video data into image data; inputting the generated image data into a first trained model to calculate a first feature vector; extracting skeletal information in time series from the first video data; inputting the extracted time-series skeletal information into a second trained model to calculate a second feature vector; combining the calculated first feature vector and second feature vector to generate a third feature vector; inputting the generated third feature vector into a third trained model to restore the second video data as an attention map; and inputting time-series RGB information constituting the first video data into a fourth trained model to calculate a score for each behavioral class, wherein the value of each channel input into the fourth trained model is multiplied by the attention map.

[0016] A program according to a third embodiment of the present invention causes a computer to execute the following steps: converting audio data out of a pair of audio data and video data into image data; inputting the generated image data into a first trained model to calculate a first feature vector; inputting time-series RGB information constituting the video data into a second trained model to calculate a second feature vector; combining the calculated first and second feature vectors to generate a third feature vector; inputting the generated third feature vector into a third trained model to generate a fourth feature vector as an attention map; extracting skeletal information in time series from the video data; inputting the extracted time-series skeletal information into a fourth trained model to calculate a score for each behavior class, wherein the value of each node of the skeleton to be input into the fourth trained model is multiplied by the attention map. Effect of the Invention

[0017] The present invention provides a robust behavior recognition technology capable of estimating behavior in a variety of situations. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram showing a configuration of a behavior recognition system 1 according to a first embodiment. [Diagram 2] FIG. 1 is a flow diagram showing a behavior recognition process according to a first embodiment; [Diagram 3] FIG. 1 shows a configuration of a behavior recognition system 10 according to a second embodiment. [Figure 4] FIG. 11 is a flow diagram showing a behavior recognition process according to the second embodiment; [Diagram 5] FIG. 13 is a diagram showing the configuration of a behavior recognition system 20 according to a third embodiment. [Figure 6] FIG. 11 is a flow diagram showing a behavior recognition process according to the third embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] 1. First embodiment 1-1.Configuration A behavior recognition system 1 according to a first embodiment of the present invention will be described with reference to the drawings. In the behavior recognition system 1 according to the present embodiment, behavior is estimated by combining features extracted by two methods, 3D convolution-based and skeleton-based, and a voice classification method, thereby enabling robust estimation that is effective in various situations. Specifically, first, two methods, 3D convolution-based and skeleton-based, are applied to one input video to extract two features with different characteristics. Next, a voice classification method is applied to the input voice to extract features. After that, the three extracted features are combined and normalized before being input to a multilayer perceptron to obtain a behavior estimation result. This behavior estimation method can capture a wider variety of features than a single method, enabling highly accurate inference.

[0020] Fig. 1 is a diagram showing the configuration of a behavior recognition system 1. As shown in the figure, the behavior recognition system 1 has a preprocessing unit 2, a first feature extraction unit 3, a second feature extraction unit 4, a skeleton information extraction unit 5, a third feature extraction unit 6, a combination unit 7, and a behavior estimation unit 8. These components are realized by a processor executing one or more programs stored in a storage device. Each component will be described below.

[0021] The pre-processing unit 2 is a means for converting the input voice into an image called a mel spectrogram. This pre-processing unit 2 first applies a short-time Fourier transform to the waveform of the input voice to extract a spectrogram. Next, a mel filter bank is applied to the generated spectrogram to extract a mel spectrogram. The extracted mel spectrogram is output to the first feature extraction unit 3.

[0022] The first feature extraction unit 3 is a means for inputting the mel spectrogram output from the preprocessing unit 2 into a trained model and calculating a feature vector. In other words, the first feature extraction unit 3 is a means for extracting the feature amount of the input voice.

[0023] The trained model used by the first feature extraction unit 3 is generated by having a convolutional neural network (hereinafter simply referred to as "CNN") train on training data that associates voice with actions. Here, CNN is a computational model used to extract features from images. This CNN is composed of a convolutional layer, a pooling layer, a fully connected layer, and an output layer. Each layer will be explained below.

[0024] The convolution layer is a means for performing convolution processing on image data. The convolution processing performed by the convolution layer is a process of repeatedly performing a multiply-and-accumulate operation between a part of the image data and the filter value of the weight filter. The convolution processing extracts image features. The image data that has been subjected to the convolution processing is converted by an activation function and then output to the pooling layer.

[0025] The pooling layer is a means for performing a pooling process on the image data output from the convolution layer. The pooling process performed by the pooling layer is a process of dividing the image data into a plurality of regions, extracting predetermined data for each region, and arranging the data in a matrix. This pooling process makes it possible to reduce the image data while retaining the features extracted by the convolution layer.

[0026] The fully connected layer is a means for judging an image using image data that has been subjected to convolution processing and pooling processing. This fully connected layer has a configuration in which all nodes in a layer are connected to all nodes in the next layer. When a two-dimensional feature map is input as image data to this fully connected layer, it is expanded into one dimension and output.

[0027] The output layer is a means for determining the image using the output data of the fully connected layer. This output layer uses a softmax function or the like as a likelihood function to calculate a score for each behavior class.

[0028] The CNN with the layers described above is capable of supervised learning. For example, backpropagation can be used for supervised learning. By having this CNN learn the training data, the filter values ​​of the weight filters and the weight coefficients of the fully connected layers can be optimized.

[0029] Of the CNNs trained in this way, only the convolutional and pooling layers are used as trained models. In other words, the fully connected and output layers are not used as trained models.

[0030] In addition, the generation of this trained model may use, for example, the PANNs technology described in the following document. Qiuqiang Kong, et al. “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition”, [online], [Retrieved December 14, 2020], Internet<https: / / arxiv.org / pdf / 1912.10211.pdf>

[0031] The first feature extraction unit 3 calculates a feature vector using the trained model thus generated. The calculated feature vector is output to the combination unit 7.

[0032] Next, the second feature extraction unit 4 will be described. The second feature extraction unit 4 is a means for inputting the time-series RGB information constituting the input video to the trained model and calculating a feature vector. That is, the second feature extraction unit 4 is a means for extracting the feature amount of the input video based on the RGB information.

[0033] The trained model used by the second feature extraction unit 4 is generated by having a three-dimensional convolutional neural network (hereinafter, simply referred to as "3D-CNN") train teacher data that associates actions with video. Here, 3D-CNN is a computational model used to extract features from video. Like CNN, this 3D-CNN is composed of a convolutional layer, a pooling layer, a fully connected layer, and an output layer. However, since this 3D-CNN receives video (in other words, video) instead of still images, the local receptive fields of the convolutional layer and pooling layer are regions that have a size in the vertical and horizontal directions as well as in the time direction.

[0034] This 3D-CNN is capable of supervised learning. For example, backpropagation can be used for supervised learning. By having this 3D-CNN learn the training data, it is possible to optimize the filter values ​​of the weight filters and the weight coefficients of the fully connected layer.

[0035] Of the 3D-CNN trained in this way, only the convolutional layer and the pooling layer are used as trained models. In other words, the fully connected layer and the output layer are not used as trained models.

[0036] In addition, to generate this trained model, for example, the SlowFast Networks technology described in the above non-patent document 1 or the I3D technology described in the above non-patent document 2 may be used.

[0037] The second feature extraction unit 4 calculates a feature vector using the trained model thus generated. The calculated feature vector is output to the combination unit 7.

[0038] Next, the skeleton information extraction unit 5 will be described. The skeletal information extraction unit 5 is a means for extracting skeletal information from the input video in a time series. The skeletal information extracted by the skeletal information extraction unit 5 is information indicating the posture of the person in the image. More specifically, it is information in which the two-dimensional coordinates of the parts of the person in the image, such as the nose, neck, both shoulders, both elbows, both wrists, both hips, both knees, and both ankles, are linked by a frame to the reliability assigned to each part. This skeletal information is extracted, for example, using the OpenPose technology described in Non-Patent Document 3 above. The extracted skeletal information is output to the third feature extraction unit 6.

[0039] The third feature extraction unit 6 is a means for inputting the time-series skeletal information output from the skeletal information extraction unit 5 to the trained model and calculating a feature vector. That is, the third feature extraction unit 6 is a means for extracting the feature amount of the input video based on the skeletal information.

[0040] The trained model used by the third feature extraction unit 6 is generated by having a graph convolutional neural network (hereinafter simply referred to as "GCN") train on training data that associates actions with time-series skeletal information. Here, a GCN is a computational model used to extract features from time-series skeletal information. This GCN is composed of a graph convolutional layer, a fully connected layer, and an output layer.

[0041] This GCN is capable of supervised learning. For example, backpropagation can be used for supervised learning. By having this GCN learn the training data, the filter values ​​of the weight filters and the weight coefficients of the fully connected layer can be optimized.

[0042] Of the GCNs trained in this way, only the graph convolutional layer is used as the trained model: in other words, the fully connected layer and the output layer are not used as the trained model.

[0043] In addition, the generation of this trained model may use, for example, the ST-GCN technique described in Non-Patent Document 4 above.

[0044] The third feature extraction unit 6 calculates a feature vector using the trained model thus generated. The calculated feature vector is output to the combination unit 7.

[0045] Next, the coupling portion 7 will be described. The combining unit 7 is a means for combining (in other words, concatenating) the feature vectors output from the first feature extraction unit 3, the second feature extraction unit 4, and the third feature extraction unit 6. The combining unit 7 combines the three feature vectors to generate one feature vector. After generating one feature vector, the combining unit 7 performs normalization processing so that the feature amounts become normally distributed, and outputs the vector to the behavior estimation unit 8. Note that, for example, Batchnorm1D is used for this normalization processing.

[0046] The behavior estimation unit 8 is a means for inputting the feature vector output from the combination unit 7 into the trained model and calculating a score for each behavior class. In other words, the behavior estimation unit 8 is a means for estimating a behavior based on two different feature amounts, that is, the feature amount of the input voice and the feature amount of the input video.

[0047] The trained model used by the behavior estimation unit 8 is generated by having a multi-layer perceptron (hereinafter simply referred to as "MLP") train on training data. As a specific training method, trained models of the first feature extraction unit 3, the second feature extraction unit 4, and the third feature extraction unit 6 are first constructed in advance. Then, training data in which actions are associated with sets of audio data and video data is repeatedly input to the behavior recognition system 1, and the weighting coefficients of the MLP are optimized.

[0048] The behavior estimation unit 8 uses the trained model thus generated to calculate a score for each behavior class. Each score calculated by the behavior estimation unit 8 indicates the probability of belonging to the behavior class.

[0049] 1-2.Operation Next, a description will be given of the behavior recognition process executed by the behavior recognition system 1. Fig. 2 is a flow diagram showing this behavior recognition process. Below, a method for estimating one behavior based on a pair of audio data and video data will be described with reference to this flow diagram.

[0050] First, when voice data is input to the behavior recognition system 1, the preprocessing unit 2 converts the input voice into a mel spectrogram (step Sa1). When a mel spectrogram is generated as a result of this conversion, the first feature extraction unit 3 inputs the generated mel spectrogram into the trained model and calculates a feature vector (step Sa2).

[0051] Next, the second feature extraction unit 4 inputs the time-series RGB information constituting the input video to the trained model and calculates a feature vector (step Sa3).

[0052] Next, the skeleton information extraction unit 5 extracts skeleton information in time series from the input video (step Sa4). When the time series skeleton information is extracted, the third feature extraction unit 6 inputs the extracted time series skeleton information to the trained model and calculates a feature vector (step Sa5).

[0053] Next, the combining unit 7 combines the three feature vectors calculated in steps Sa2, Sa3, and Sa5 to generate one feature vector (step Sa6). Then, the combining unit 7 performs a normalization process on the generated feature vector (step Sa7). After performing the normalization process, the behavior inferring unit 8 inputs the feature vector into the trained model and calculates a score for each behavior class (step Sa8). This concludes the description of the behavior recognition process.

[0054] According to the above-described behavior recognition system 1, behavior estimation is performed by combining feature amounts extracted by two methods, 3D convolution-based and skeleton-based. Therefore, it is possible to identify both differences in behavior due to the involvement of surrounding objects and differences in fine behavior. For example, as an example of the former behavior, it is possible to identify the behavior of drinking different types of bottled water. This is because the two behaviors can be distinguished by the color of the label attached to the bottle, i.e., RGB information. Also, as an example of the latter behavior, it is possible to identify the behavior of simply looking into a PC monitor and the behavior of facing the PC monitor and operating the keyboard.

[0055] In addition, according to the above-mentioned behavior recognition system 1, behavior estimation is performed taking into consideration the feature amount extracted by the voice classification method. Therefore, it is possible to successfully identify behaviors where voice is important. For example, it is possible to distinguish the difference between a fight and a conversation. In addition, it is possible to improve the accuracy of identifying behaviors that can be identified from a video. For example, it is possible to improve the accuracy of identifying the difference between a walking behavior and a running behavior, or the difference between a touching behavior and a hitting behavior.

[0056] In summary, the above behavior recognition system 1 enables robust behavior recognition that can estimate behavior in a variety of situations.

[0057] 2. Second embodiment 2-1.Configuration A behavior recognition system 10 according to a second embodiment of the present invention will be described with reference to the drawings. In the behavior recognition system 10 according to this embodiment, similarly to the first embodiment, behavior estimation is performed by combining features extracted by two methods, 3D convolution-based and skeleton-based, and a voice classification method. However, the method of combining features is different from that of the first embodiment.

[0058] Fig. 3 is a diagram showing the configuration of a behavior recognition system 10 according to this embodiment. As shown in the figure, the behavior recognition system 10 has a preprocessing unit 2, a first feature extraction unit 3, a skeleton information extraction unit 5, a third feature extraction unit 6, a combination unit 11, an image restoration unit 12, and a behavior estimation unit 13. These components are realized by executing one or more programs stored in a storage device by a processor. Each component will be described below.

[0059] As described in the first embodiment, the pre-processing unit 2 is a means for converting an input voice into a mel spectrogram. The mel spectrogram generated by the pre-processing unit 2 is output to the first feature extraction unit 3.

[0060] As described in the first embodiment, the first feature extraction unit 3 is a means for inputting the mel spectrogram output from the preprocessing unit 2 to the trained model and calculating a feature vector. The feature vector calculated by the first feature extraction unit 3 is output to the combining unit 11.

[0061] As described in the first embodiment, the skeleton information extraction unit 5 is a means for extracting skeleton information from an input video in a time series. The skeleton information extracted by the skeleton information extraction unit 5 is output to the third feature extraction unit 6.

[0062] As described in the first embodiment, the third feature extraction unit 6 is a means for inputting the time-series skeletal information output from the skeletal information extraction unit 5 to the trained model and calculating a feature vector. The feature vector calculated by the third feature extraction unit 6 is output to the combination unit 11.

[0063] The combining unit 11 is a means for combining (in other words, concatenating) the feature vectors output from the first feature extraction unit 3 and the third feature extraction unit 6. This combining unit 11 combines two feature vectors to generate one feature vector. The generated feature vector is output to the video restoration unit 12.

[0064] The video restoration unit 12 is a means for inputting the feature vector output from the combination unit 11 into a trained model to restore the video data. In other words, the video restoration unit 12 is a means for restoring a video reflecting the features of the input audio and the input video, based on these features.

[0065] The trained model used by the video restoration unit 12 is generated by having a generative adversarial network (hereinafter simply referred to as "GAN") learn training data. Here, a GAN is a generative model used to restore video data from a feature vector. This GAN is composed of two networks: a generator and a classifier.

[0066] Of the two networks that make up this GAN, the generator is a network that takes latent variables as input, performs deconvolution processing (in other words, transposed convolution processing) on ​​the latent variables, and outputs image data. On the other hand, the discriminator is a network that takes either the image data generated by the generator or training data as input, and discriminates whether the input data is training data or not.

[0067] Of these networks, the classifier is trained to be able to make correct classifications. On the other hand, the generator is trained to be able to generate image data that is difficult for the classifier to distinguish from the training data. By alternating between these two learning processes, a generator is generated that can generate image data that is difficult to distinguish from the training data.

[0068] Of these trained generators and classifiers, only the generator is used as the trained model, i.e., the trained classifier is not used as the trained model.

[0069] In addition, the generation of this trained model may be performed using, for example, the 3D-GAN technology described in the following document. Jiajun Wu, et al. “Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling”, [online], [Retrieved December 14, 2020], Internet<https: / / arxiv.org / pdf / 1610.07584.pdf>

[0070] The video restoration unit 12 generates video data using the trained model thus generated. The generated video data is output to the behavior estimation unit 13 as an attention map. Here, the attention map is data indicating an area in an input video that should be noted. For example, if the input video is an image of a runner, this attention map is data indicating the runner's hands and feet as areas that should be noted. The behavior estimation unit 13 uses this attention map to improve the accuracy of feature extraction.

[0071] Next, the behavior inferrer 13 will be described. The behavior estimation unit 13 is a means for inputting time-series RGB information constituting the input video into a trained model and calculating a score for each behavior class. In other words, the behavior estimation unit 13 is a means for estimating behavior based on the feature amount of the input video. The trained model used by the behavior estimation unit 13 is generated by having the 3D-CNN train the teacher data. A method for generating the trained model will be described later.

[0072] When inputting the time-series RGB information to the trained model, the behavior estimation unit 13 multiplies the value of each channel by the attention map output from the video restoration unit 12. The calculation formula at that time is shown below.

[0073] T'(x,y,z)=T(x,y,z)*(M(x,y,z)+1) …(1)

[0074] In this formula (1), T(x, y, z) represents a channel value, M(x, y, z) represents an attention map value corresponding to the channel value, and T'(x, y, z) represents a corrected channel value as a result of multiplication. The behavior estimation unit 13 improves the accuracy of feature extraction by multiplying each channel value by the attention map as shown in this formula (1).

[0075] The attention map is described in, for example, the following document (see, in particular, formula (3) and FIG. 3): Fei Wang, et al. “Residual Attention Network for Image Classification”, [online], [Retrieved December 14, 2020], Internet<https: / / arxiv.org / pdf / 1704.06904.pdf> In this document, the attention mechanism is applied to two-dimensional images, but in the processing of the behavior estimation unit 13, this attention mechanism is applied to three-dimensional video images.

[0076] Next, a method for generating a trained model will be described. To generate a trained model, first, trained models of the first feature extraction unit 3 and the third feature extraction unit 6 are constructed in advance. Then, training data in which actions are associated with sets of audio data and video data is repeatedly input to the action recognition system 10, and the filter values ​​of the weight filters of the 3D-CNN and the weight coefficients of the fully connected layer are optimized.

[0077] In addition, to generate this trained model, for example, the SlowFast Networks technology described in the above non-patent document 1 or the I3D technology described in the above non-patent document 2 may be used.

[0078] The behavior estimation unit 13 calculates a score for each behavior class using the trained model thus generated. Each score calculated by the behavior estimation unit 13 indicates the probability of belonging to the behavior class.

[0079] 2-2.Operation Next, a description will be given of the behavior recognition process executed by the behavior recognition system 10. Fig. 4 is a flow diagram showing this behavior recognition process. Below, a method for estimating one behavior based on a pair of audio data and video data will be described with reference to this flow diagram.

[0080] First, when voice data is input to the behavior recognition system 10, the preprocessing unit 2 converts the input voice into a mel spectrogram (step Sb1). When a mel spectrogram is generated as a result of this conversion, the first feature extraction unit 3 inputs the generated mel spectrogram into the trained model and calculates a feature vector (step Sb2).

[0081] Next, the skeleton information extraction unit 5 extracts skeleton information from the input video in a time series (step Sb3). When the time series skeleton information is extracted, the third feature extraction unit 6 inputs the extracted time series skeleton information to the trained model and calculates a feature vector (step Sb4).

[0082] Next, the combining unit 11 combines the two feature vectors calculated in steps Sb2 and Sb4 to generate one feature vector (step Sb5). After the feature vector is generated, the video restoration unit 12 inputs the generated feature vector into the trained model to generate an attention map (step Sb6).

[0083] Next, the behavior estimation unit 13 inputs the time-series RGB information constituting the input video to the trained model and calculates a score for each behavior class (step Sb7). At that time, the behavior estimation unit 13 multiplies the value of each channel input to the trained model by the attention map output from the video restoration unit 12. This concludes the description of the behavior recognition process.

[0084] According to the behavior recognition system 10 described above, behavior estimation is performed by combining feature amounts extracted by two methods, 3D convolution-based and skeleton-based. Therefore, it is possible to distinguish both differences in behavior due to the involvement of surrounding objects and subtle differences in behavior. In addition, according to this behavior recognition system 10, behavior estimation is performed taking into account feature amounts extracted by a voice classification method. Therefore, it is possible to successfully distinguish behaviors in which voice is important. In summary, according to this behavior recognition system 10, robust behavior recognition that can estimate behavior in a variety of situations is possible.

[0085] 3. Third embodiment 3-1.Configuration A behavior recognition system 20 according to a third embodiment of the present invention will be described with reference to the drawings. In the behavior recognition system 20 according to this embodiment, similarly to the first and second embodiments, behavior estimation is performed by combining features extracted by two methods, 3D convolution-based and skeleton-based, and a voice classification method. However, the method of combining features is different from the first and second embodiments.

[0086] Fig. 5 is a diagram showing the configuration of a behavior recognition system 20 according to this embodiment. As shown in the figure, the behavior recognition system 20 has a preprocessing unit 2, a first feature extraction unit 3, a second feature extraction unit 4, a combining unit 21, a fourth feature extraction unit 22, a skeleton information extraction unit 5, and a behavior estimation unit 23. These components are realized by a processor executing one or more programs stored in a storage device. Each component will be described below.

[0087] As described in the first embodiment, the pre-processing unit 2 is a means for converting an input voice into an image called a mel spectrogram. The mel spectrogram generated by the pre-processing unit 2 is output to the first feature extraction unit 3.

[0088] As described in the first embodiment, the first feature extraction unit 3 is a means for inputting the mel spectrogram output from the preprocessing unit 2 to a trained model and calculating a feature vector. The feature vector calculated by the first feature extraction unit 3 is output to the combining unit 21.

[0089] As described in the first embodiment, the second feature extraction unit 4 is a means for inputting time-series RGB information constituting an input video to a trained model and calculating a feature vector. The feature vector calculated by the second feature extraction unit 4 is output to the combination unit 21.

[0090] The combining unit 21 is a means for combining (in other words, concatenating) the feature vectors output from the first feature extraction unit 3 and the second feature extraction unit 4. This combining unit 21 combines two feature vectors to generate one feature vector. The generated feature vector is output to the fourth feature extraction unit 22.

[0091] The fourth feature extraction unit 22 is a means for inputting the feature vector output from the combining unit 21 into the trained model and calculating another feature vector. The trained model used for calculating the feature vector by the fourth feature extraction unit 22 is generated by having the MLP train the teacher data. As a specific training method, trained models of the first feature extraction unit 3 and the second feature extraction unit 4 are first constructed in advance. Then, the pre-processing unit 2, the first feature extraction unit 3, the second feature extraction unit 4, the combining unit 21, and the fourth feature extraction unit 22 are linked as shown in FIG. 5. Then, the teacher data in which actions are associated with sets of audio data and video data is repeatedly input into this linked system to optimize the weight coefficients of the MLP.

[0092] The fourth feature extraction unit 22 calculates a feature vector using the trained model thus generated. The calculated feature vector is output as an attention map to the behavior estimation unit 23. The behavior estimation unit 23 improves the accuracy of feature extraction by using this attention map.

[0093] Next, the skeleton information extraction unit 5 will be described. As described in the first embodiment, the skeletal information extraction unit 5 is a means for extracting skeletal information from an input video in a time series. The skeletal information extracted by the skeletal information extraction unit 5 is output to the behavior estimation unit 23.

[0094] The behavior estimation unit 23 is a means for inputting the time-series skeletal information output from the skeletal information extraction unit 5 into the trained model and calculating a score for each behavior class. In other words, the behavior estimation unit 23 is a means for estimating behavior based on the feature amount of the input video. The trained model used by the behavior estimation unit 13 is generated by having the GCN train the teacher data. A method for generating the trained model will be described later.

[0095] When inputting the time-series skeleton information to the trained model, the behavior estimation unit 23 multiplies the value of each node of the skeleton by the attention map output from the fourth feature extraction unit 22. The calculation formula at that time is shown below.

[0096] T'(x)=T(x)*(M(x)+1) …(2)

[0097] In this formula (2), T(x) represents the value of the skeleton node, M(x) represents the attention map value corresponding to the node value, and T'(x) represents the corrected node value as a result of the multiplication. The behavior estimation unit 23 improves the accuracy of feature extraction by multiplying the value of each skeleton node by the attention map as shown in this formula (2).

[0098] The attention map is described in the above-mentioned document "Residual Attention Network for Image Classification." In this document, the attention mechanism is applied to two-dimensional images, but in the processing of the behavior estimation unit 23, this attention mechanism is applied to one-dimensional vectors.

[0099] Next, a method for generating a trained model will be described. To generate a trained model, first, trained models of the first feature extraction unit 3, the second feature extraction unit 4, and the fourth feature extraction unit 22 are constructed in advance. Then, training data in which actions are associated with sets of audio data and video data is repeatedly input to the action recognition system 20, and the filter values ​​of the GCN weight filters and the weight coefficients of the fully connected layer are optimized.

[0100] In addition, the generation of this trained model may use, for example, the ST-GCN technique described in Non-Patent Document 4 above.

[0101] The behavior estimation unit 23 calculates a score for each behavior class using the trained model thus generated. Each score calculated by the behavior estimation unit 23 indicates the probability of belonging to the behavior class.

[0102] 3-2.Operation Next, a description will be given of the behavior recognition process executed by the behavior recognition system 20. Fig. 6 is a flow diagram showing this behavior recognition process. Below, a method for estimating one behavior based on a pair of audio data and video data will be described with reference to this flow diagram.

[0103] First, when voice data is input to the behavior recognition system 10, the preprocessing unit 2 converts the input voice into a mel spectrogram (step Sc1). When a mel spectrogram is generated as a result of this conversion, the first feature extraction unit 3 inputs the generated mel spectrogram into the trained model and calculates a feature vector (step Sc2).

[0104] Next, the second feature extraction unit 4 inputs the time-series RGB information constituting the input video to the trained model and calculates a feature vector (step Sc3).

[0105] Next, the combining unit 21 combines the two feature vectors calculated in steps Sc2 and Sc3 to generate one feature vector (step Sc4). After the feature vector is generated, the fourth feature extracting unit 22 inputs the generated feature vector into the trained model to generate an attention map (step Sc5).

[0106] Next, the skeleton information extraction unit 5 extracts skeleton information from the input video in a time series (step Sc6). When the time series skeleton information is extracted, the behavior estimation unit 23 inputs the extracted time series skeleton information to the trained model and calculates a score for each behavior class (step Sc7). At that time, the behavior estimation unit 23 multiplies the value of each skeleton node to be input to the trained model by the attention map output from the fourth feature extraction unit 22. This concludes the description of the behavior recognition process.

[0107] According to the behavior recognition system 20 described above, behavior estimation is performed by combining feature amounts extracted by two methods, 3D convolution-based and skeleton-based. Therefore, it is possible to distinguish both differences in behavior due to the involvement of surrounding objects and differences in fine behavior. In addition, according to this behavior recognition system 20, behavior estimation is performed taking into account feature amounts extracted by a voice classification method. Therefore, it is possible to successfully distinguish behaviors in which voice is important. In summary, according to the behavior recognition system 20, robust behavior recognition that can estimate behavior in a variety of situations is possible.

[0108] 4. Variations In the behavior recognition process according to each of the above embodiments, the order of each step may be interchanged as long as no inconsistency occurs. For example, in the behavior recognition process according to the first embodiment, the set of steps Sa1 and Sa2 may be interchanged with the set of step Sa3 and the set of steps Sa4 and Sa5. As another example, in the behavior recognition process according to the second embodiment, the set of steps Sb1 and Sb2 may be interchanged with the set of steps Sb3 and Sb4. As yet another example, in the behavior recognition process according to the third embodiment, the set of steps Sc1 and Sc2 may be interchanged with the set of step Sc3. [Explanation of symbols]

[0109] Reference Signs List 1, 10, 20... Behavior recognition system, 2... Preprocessing unit, 3... First feature extraction unit, 4... Second feature extraction unit, 5... Skeleton information extraction unit, 6... Third feature extraction unit, 7, 11, 21... Combination unit, 8, 13, 23... Behavior estimation unit, 12... Video restoration unit, 22... Fourth feature extraction unit

Claims

1. a pre-processing unit that converts the audio data out of a pair of audio data and first video data into image data; a first feature extraction unit that inputs the generated image data into a first trained model to calculate a first feature vector; a skeleton information extracting unit that extracts skeleton information in time series from the first video data; a second feature extraction unit that inputs the extracted time-series skeletal information into a second trained model and calculates a second feature vector; a combining unit that combines the calculated first feature vector and the calculated second feature vector to generate a third feature vector; an image restoration unit that inputs the generated third feature vector into a third trained model to restore the second image data as an attention map; The time-series RGB information constituting the first video data is input to a fourth trained model. a behavior estimation unit that calculates a score for each behavior class, the behavior estimation unit multiplying the value of each channel to be input to the fourth trained model by the attention map; An activity recognition system comprising:

2. a pre-processing unit that converts audio data out of a pair of audio data and video data into image data; a first feature extraction unit that inputs the generated image data into a first trained model to calculate a first feature vector; a second feature extraction unit that inputs time-series RGB information constituting the video data into a second trained model and calculates a second feature vector; a combining unit that combines the calculated first feature vector and the calculated second feature vector to generate a third feature vector; a third feature extraction unit that inputs the generated third feature vector into a third trained model to generate a fourth feature vector as an attention map; a skeletal information extraction unit that extracts skeletal information from the video data in time series; a behavior estimation unit that inputs the extracted time-series skeleton information into a fourth trained model and calculates a score for each behavior class, the behavior estimation unit multiplying a value of each node of the skeleton to be input into the fourth trained model by the attention map; An activity recognition system comprising:

3. converting the audio data of a pair of audio data and first video data into image data; inputting the generated image data into a first trained model to calculate a first feature vector; extracting skeletal information from the first video data in time series; inputting the extracted time-series skeletal information into a second trained model and calculating a second feature vector; combining the calculated first feature vector and the calculated second feature vector to generate a third feature vector; inputting the generated third feature vector into a third trained model to restore the second video data as an attention map; a step of inputting time-series RGB information constituting the first video data into a fourth trained model and calculating a score for each behavior class, the step multiplying the value of each channel inputted into the fourth trained model by the attention map; The activity recognition method includes:

4. converting audio data from a pair of audio data and video data into image data; inputting the generated image data into a first trained model to calculate a first feature vector; inputting time-series RGB information constituting the video data into a second trained model and calculating a second feature vector; combining the calculated first feature vector and the calculated second feature vector to generate a third feature vector; inputting the generated third feature vector into a third trained model to generate a fourth feature vector as an attention map; extracting skeletal information from the video data in time series; a step of inputting the extracted time-series skeleton information into a fourth trained model and calculating a score for each behavior class, the step of multiplying the value of each node of the skeleton inputted into the fourth trained model by the attention map; The activity recognition method includes:

5. On the computer, converting the audio data of a pair of audio data and first video data into image data; inputting the generated image data into a first trained model to calculate a first feature vector; extracting skeletal information from the first video data in time series; inputting the extracted time-series skeletal information into a second trained model and calculating a second feature vector; combining the calculated first feature vector and the calculated second feature vector to generate a third feature vector; inputting the generated third feature vector into a third trained model to restore the second video data as an attention map; a step of inputting time-series RGB information constituting the first video data into a fourth trained model and calculating a score for each behavior class, the step multiplying the value of each channel inputted into the fourth trained model by the attention map; A program for executing the above.

6. On the computer, converting audio data from a pair of audio data and video data into image data; inputting the generated image data into a first trained model to calculate a first feature vector; inputting time-series RGB information constituting the video data into a second trained model and calculating a second feature vector; combining the calculated first feature vector and the calculated second feature vector to generate a third feature vector; inputting the generated third feature vector into a third trained model to generate a fourth feature vector as an attention map; extracting skeletal information from the video data in time series; The extracted time-series skeleton information is input to a fourth trained model, and the fourth trained model is trained for each class of behavior. a step of multiplying the value of each node of a skeleton to be input to the fourth trained model by the attention map; A program for executing the above.

Images