Motion identification learning device, motion identification device, learning method, and program

Abstract

To learn a motion identification model generalized for a variety of people using a biased dataset.SOLUTION: A motion identification unit obtains an estimation motion class attribution probability of motion input data obtained from person motion data, and a person identification unit obtains an estimation person class attribution probability of person input data obtained from person input data. A hostile identification unit calculates a hostile person class attribution probability indicating an estimated probability that the motion input data is attributed to each person class, and a hostile motion class attribution probability indicating an estimated probability that the person input data is attributed to each motion class. A loss evaluation unit calculates a loss evaluation value on the basis of the estimation motion class attribution probability, the estimation person class attribution probability, the hostile motion class attribution probability, the hostile person class attribution probability, a given teacher motion class attribution probability, and a given teacher person class attribution probability. A learning unit learns parameter values that are used for respective processes at the motion identification unit, the person identification unit, and the hostile identification unit on the basis of the loss evaluation value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a motion identification learning device, a motion identification device, a learning method, and a program. [Background technology]

[0002] In recent years, with the spread of sensor equipment such as motion capture devices and wearable devices, and the improvement of image recognition technology, it has become easier to sense human movements. Data that can be recorded by these sensors include multi-channel time-series signal data recorded by accelerometers and gyro sensors mounted on smartphones and wearable watches, and human skeletal time-series data recorded by infrared sensors and markers attached to the body. Human movement identification using this data is an important technology with a wide range of applications, such as human authentication, life log applications, VR (virtual reality) / AR (augmented reality) applications, and automatic surveillance systems.

[0003] Recognizing human actions using sensed data is often achieved by using a predictive model based on machine learning. Compared to heuristic methods, methods using machine learning have the advantage of being more versatile. On the other hand, this method has the problem that the accuracy of action recognition is low for data of people other than the person used for learning. This is because each person's actions vary from person to person, and the same action can differ greatly in the data. For example, when considering the action of walking, the data can differ greatly depending on the walking speed and the magnitude of arm swing, and as a result, it may not be possible to properly recognize another person's walking action.

[0004] In order to solve such problems, various technologies have been disclosed. For example, there is a technology that avoids the recognition degradation problem due to individual differences by preparing a prediction model for each target person individually (for example, see Non-Patent Document 1). There is also a method that detects a person similar to the target person from a large number of accumulated motion data of people and uses a prediction model trained with the data of that person (for example, see Patent Document 1). There is also a technology that uses a generative model trained with data of other people to complement data when data necessary for learning a prediction model for a target person is partially missing (for example, see Non-Patent Document 2). There is also a method that uses active learning and transfer learning to adapt a prediction model to people other than those included in the learning examples (for example, see Non-Patent Document 3 and Non-Patent Document 4). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-248017 A [Non-patent literature]

[0006] [Non-Patent Document 1] Gary M. Weiss and Jeffrey W. Lockhart, "The impact of personalization on smartphone-based activity recognition," In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence. [Non-Patent Document 2] N. Suzuki, Y. Watanabe, and A. Nakazawa. 2020. GAN-based Style Transformation to Improve Gesture-recognition Accuracy. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 154 (December 2020), 20 pages. [Non-Patent Document 3] R. Fallahzadeh and H. Ghasemzadeh. 2017. Personalization without User Interruption: Boosting Activity Recognition in New Subjects Using Unlabeled Data. In 2017 ACM / IEEE 8th International Conference on Cyber-Physical Systems (ICCPS). 293-302. [Non-Patent Document 4] Seyed Ali Rokni, Marjan Nourollahi, and Hassan Ghasemzadeh. 2018. Personalized Human Activity Recognition Using Convolutional Neural Networks. In Thirty-Second AAAI Conference on Artificial Intelligence. Summary of the Invention [Problem to be solved by the invention]

[0007] In order to achieve highly accurate action recognition that is not affected by individual differences and generalizes to a variety of people, the various techniques mentioned above have been proposed, but these techniques require a high-quality training dataset.

[0008] First, in all of the techniques in Patent Document 1, Non-Patent Document 1, Non-Patent Document 3, and Non-Patent Document 4, it is required that the data for each person in the training dataset includes data for all action classes evenly, and that there are no action classes with missing data (hereinafter referred to as "missing action classes").

[0009] Non-Patent Document 2 discloses a technology for generating and complementing data of a missing action class using a generative model even if the target person's data has a missing action class when learning an action identification model for the target person. However, in order to learn a generative model for complementation, it is necessary to prepare a training data set in which no missing action class exists for any person other than the target person.

[0010] As outlined above, the currently disclosed technology requires a high-quality training data set. However, due to constraints such as the development cost of the recognition model, it is often the case that only a biased, low-quality data set with missing action classes can be prepared as data for each person. In such cases, the currently disclosed technology cannot achieve high-precision action recognition that generalizes to a variety of people.

[0011] In consideration of the above circumstances, the present invention aims to provide an action recognition learning device, an action recognition device, a learning method, and a program that are capable of learning an action recognition model that generalizes to a variety of people, even when only a biased data set in which each person's data has missing action classes can be prepared. [Means for solving the problem]

[0012] One aspect of the present invention includes a motion input preprocessing unit that generates motion input data representing motion-related features from human motion data that is data representing a human motion, a human input preprocessing unit that generates human input data representing motion-related features from the human motion data, a motion identification unit that calculates an estimated motion class belonging probability that is an estimated probability that the motion input data belongs to each of one or more motion classes, a human identification unit that calculates an estimated person class belonging probability that is an estimated probability that the human input data belongs to each of one or more person classes, an adversarial identification unit that calculates an adversarial person class belonging probability that is an estimated probability that the motion input data belongs to each of one or more person classes and an adversarial motion class belonging probability that is an estimated probability that the human input data belongs to a motion class, and a loss evaluation unit that calculates a loss evaluation value that represents an estimation error based on a probability, the hostile action class membership probability, the hostile person class membership probability, a given teacher action class membership probability indicating the probability that the human action data belongs to each of the action classes, and a given teacher person class membership probability indicating the probability that the human action data belongs to each of the person classes; and a learning unit that learns, based on the loss evaluation value, a value of a first parameter used by the action recognition unit in a process for generating the action input data, a value of a second parameter used by the person recognition unit in a process for generating the person input data, a value of a third parameter used by the adversarial recognition unit in a process for calculating the hostile person class membership probability, and a value of a fourth parameter used by the adversarial recognition unit in a process for calculating the hostile action class membership probability.

[0013] One aspect of the present invention is the above-mentioned action identification learning device, wherein the action input pre-processing unit generates the action input data by performing processing on the person action data to cause information that identifies a person to be lost, and the person input pre-processing unit generates the person input data by performing processing on the person action data to cause information that identifies a action to be lost.

[0014] One aspect of the present invention is the above-mentioned action identification learning device, wherein the action identification unit comprises a action feature extractor that extracts action features from the action input data, and a action classifier that outputs the estimated action class membership probability from the action features, the person identification unit comprises a person feature extractor that extracts person features from the action input data, and a person classifier that outputs the estimated person class membership probability from the person features, and the adversarial identification unit calculates the adversarial person class membership probability based on the action features extracted by the action feature extractor, and calculates the adversarial action class membership probability based on the person features extracted by the person feature extractor.

[0015] One aspect of the present invention is the above-mentioned action identification learning device, wherein the adversarial identification unit includes a shared processor that performs processing on the action features and the person features to remove information unnecessary for action identification and person identification, an adversarial person classifier that calculates an adversarial person class belonging probability based on the action features processed by the shared processor, and an adversarial action classifier that calculates an adversarial action class belonging probability based on the person features processed by the shared processor.

[0016] One aspect of the present invention is the above-mentioned action recognition learning device, wherein the loss evaluation unit is configured to evaluate an action class recognition loss evaluation value that becomes smaller as the similarity between the estimated action class membership probability and the teacher action class membership probability becomes higher, a person class recognition loss evaluation value that becomes smaller as the similarity between the estimated person class membership probability and the teacher person class membership probability becomes higher, an adversarial action class recognition loss evaluation value that becomes smaller as the similarity between the adversarial action class membership probability and the teacher action class membership probability becomes higher, an adversarial person class recognition loss evaluation value that becomes smaller as the similarity between the adversarial person class membership probability and the teacher person class membership probability becomes higher, and a loss evaluation value that indicates ambiguity of the adversarial action class recognition probability. the learning unit calculates a first entropy value indicating the ambiguity of the hostile person class membership probability and a second entropy value indicating the ambiguity of the hostile person class membership probability, and the learning unit learns the value of the first parameter so as to reduce the action class identification loss evaluation value and increase the hostile person class identification loss evaluation value and the first entropy value, learns the value of the second parameter so as to reduce the person class identification loss evaluation value and increase the hostile action class identification loss evaluation value and the second entropy value, and learns the values ​​of the third parameter and the fourth parameter so as to reduce the hostile action class identification loss evaluation value and the hostile person class identification loss evaluation value.

[0017] One aspect of the present invention is a motion identification device including a motion identification unit that calculates a probability of human motion data belonging to a motion class by using the value of the first parameter learned by the above-mentioned motion identification learning device.

[0018] One aspect of the present invention includes a motion input preprocessing step of generating motion input data representing motion-related features from human motion data, which is data representing a human motion; a human input preprocessing step of generating human input data representing motion-related features from the human motion data; a motion identification step of calculating an estimated motion class membership probability, which is an estimated probability that the motion input data belongs to each of one or more motion classes; a human identification step of calculating an estimated person class membership probability, which is an estimated probability that the human input data belongs to each of one or more person classes; an adversarial identification step of calculating an adversarial person class membership probability, which is an estimated probability that the motion input data belongs to each of one or more person classes, and an adversarial motion class membership probability, which is an estimated probability that the human input data belongs to a motion class; a loss evaluation step of calculating a loss evaluation value representing an estimation error based on an action class membership probability, the hostile person class membership probability, a given teacher action class membership probability indicating the probability that the human action data belongs to each of the action classes, and a given teacher person class membership probability indicating the probability that the human action data belongs to each of the person classes; and a learning step of learning, based on the loss evaluation value, a first parameter value used in a process of generating the action input data in the action identification step, a second parameter value used in a process of generating the person input data in the person identification step, a third parameter value used in a process of calculating the hostile person class membership probability in the adversarial identification step, and a fourth parameter value used in a process of calculating the hostile action class membership probability in the adversarial identification step.

[0019] One aspect of the present invention is a program for causing a computer to function as the above-described action discrimination learning device. Effect of the Invention

[0020] According to the present invention, it is possible to learn an action recognition model that generalizes to a variety of people, even from a biased data set in which data for each person contains missing action classes. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a functional block diagram illustrating an example of an action discrimination learning device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram showing an example of the configuration of an action recognition unit, a person recognition unit, an adversarial recognition unit, and a loss assessment unit according to the present embodiment. [Diagram 3] 1 is a functional block diagram illustrating an example of an action identification device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a functional block diagram showing an example of the operation of the action discrimination learning device according to the present embodiment. [Diagram 5] FIG. 2 is a schematic diagram showing a hardware configuration common to each device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0023] <Summary> First, an overview of the present embodiment will be described. The present embodiment provides a technique that can learn an action recognition model even when a training data set in which missing action classes exist for each person is used.

[0024] [Example of a dataset with missing action classes for each person] First, an example in which a missing action class exists for each person will be given. The dataset contains two people, person A and person B, and four types of actions, action 1, action 2, action 3, and action 4. However, for person A, the dataset contains data for action 1 and action 2, but does not contain data for action 3 and action 4, and for person B, the dataset contains data for action 3 and action 4, but does not contain data for action 1 and action 2. When such a training dataset is given, an action identification model trained by a general learning technique depends on data features due to individual differences in actions, and may significantly impair recognition accuracy due to erroneous recognition. Specifically, since the training dataset contains only action 1 and action 2 for person A's actions, for example, even if data for action 3 and action 4 of person A are input, a problem occurs in which the data is erroneously recognized as action 1 or action 2 depending on the characteristics of person A.

[0025] In this embodiment, when the above-mentioned training data set is given, a prediction model is trained that can appropriately identify actions even when the data includes a pair of a person and an action that does not exist in the training data set. This embodiment deals with the problem of the existence of missing action classes for each person by performing learning that separates action-dependent features and person-dependent features of the data.

[0026] <Configuration example of motion discrimination learning device> FIG. 1 is a functional block diagram showing an example of an action identification learning device 100 according to the present embodiment. The action identification learning device 100 is realized by using an information processing device such as a personal computer or a server device. The action identification learning device 100 includes a control unit 90 and a learning result storage unit 120. The control unit 90 is configured by using a processor such as a CPU (Central Processing Unit) and a memory. The control unit 90 functions as an action input preprocessing unit 101, a person input preprocessing unit 102, an action identification unit 103, a person identification unit 104, an adversarial identification unit 105, a loss evaluation unit 106, and a learning unit 107 by the processor executing a program. Note that all or a part of the functions of the control unit 90 may be realized by using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded in a computer-readable storage medium. The computer-readable storage medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a semiconductor storage device (for example, an SSD: Solid State Drive), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunication line.

[0027] The motion discrimination learning device 100 and the data storage unit 110 are communicatively connected. The data storage unit 110 is configured using a device capable of reading data from a storage device such as a magnetic hard disk device or a semiconductor storage device, or a recording medium such as a CD-ROM. The motion discrimination learning device 100 reads a human motion data set from the data storage unit 110. The human motion data set includes a plurality of human motion data. The human class is, for example, human A, human B, ..., and the motion class is, for example, motion 1, motion 2, ..., etc. For example, the human motion data is data representing the time-series positions of feature points such as joints when a person is performing a motion. In addition to the positions of the feature points, information such as acceleration may be included.

[0028] The motion input preprocessing unit 101 receives a set of human motion data, performs preprocessing on each human motion data to obtain motion-related features, and generates motion input data. The motion input preprocessing unit 101 outputs the generated motion input data to the motion identification unit 103 and the adversarial identification unit 105.

[0029] The person input preprocessing unit 102 receives a set of person motion data, performs preprocessing on each person motion data to obtain characteristics related to a person, and generates person input data. The person input preprocessing unit 102 outputs the generated person input data to the person identification unit 104 and the adversarial identification unit 105.

[0030] FIG. 2 is a functional block diagram showing an example of the configuration of the action identifying unit 103, the person identifying unit 104, the adversarial identifying unit 105, and the loss evaluating unit 106 according to this embodiment.

[0031] The action identification unit 103 is composed of a action feature extractor 1031 and an action identifier 1032. The action identification unit 103 receives action input data from the action input preprocessing unit 101.

[0032] The motion feature extractor 1031 outputs a motion feature of the motion input data. The motion feature is a numerical vector that represents the motion feature of the motion input data. The motion feature is expressed as a vector having n1-dimensional elements determined by a numerical value n1. The motion feature extractor 1031 operates based on a function having parameters that can output such a motion feature. The motion feature extractor 1031 reads the parameter values ​​stored in the learning result storage unit 120, inputs the motion input data to a function that uses the read parameter values, and outputs the motion feature.

[0033] The action classifier 1032 outputs the estimated action class belonging probability of the action input data. The estimated action class belonging probability is an estimation result of the probability that the human action data belongs to each action class. The action classifier 1032 operates based on a function having parameters capable of outputting such an estimated action class belonging probability. The action classifier 1032 reads the parameter values ​​stored in the learning result storage unit 120, inputs the action feature amount to a function using the read parameter values, and outputs the estimated action class belonging probability.

[0034] The person identification unit 104 is composed of a person feature extractor 1041 and a person classifier 1042. The person identification unit 104 receives person input data from the person input preprocessing unit .

[0035] The person feature extractor 1041 outputs a person feature amount of the person input data. The person feature amount is a numerical vector that represents the features of a person in the person input data. The person feature amount is represented by a vector having n1-dimensional elements determined by a numerical value n1. The person feature extractor 1041 operates based on a function having parameters that can output such person feature amounts. The person feature extractor 1041 reads the parameter values ​​stored in the learning result storage unit 120, inputs the action input data to a function that uses the read parameter values, and outputs the person feature amount of the person input data.

[0036] The person classifier 1042 outputs an estimated person class belonging probability of the person input data. The estimated person class belonging probability is an estimation result of the probability that the person action data belongs to each person class. The person classifier 1042 operates based on a function having parameters capable of outputting such an estimated person class belonging probability. The person classifier 1042 reads the parameter values ​​stored in the learning result storage unit 120, inputs the person feature amount to a function using the read parameter values, and outputs the estimated person class belonging probability.

[0037] The adversarial classification unit 105 is composed of a motion feature extractor 1031, a person feature extractor 1041, and a shared adversarial classifier 1051. The adversarial classification unit 105 receives motion input data from the motion input preprocessing unit 101. The shared adversarial classifier 1051 receives motion features extracted from the motion input data by the motion feature extractor 1031 as input, and outputs an adversarial person class belonging probability. The adversarial person class belonging probability is a person class belonging probability of the person motion data estimated from the motion features. In addition, the adversarial classification unit 105 receives person input data from the person input preprocessing unit 102. The shared adversarial classifier 1051 receives person features extracted from the person input data by the person feature extractor 1041 as input, and outputs an adversarial motion class belonging probability. The adversarial motion class belonging probability is a motion class belonging probability of the person motion data estimated from the person features. The shared adversarial classifier 1051 operates based on a function having parameters capable of outputting such a hostile person class belonging probability and a hostile action class belonging probability. The shared adversarial classifier 1051 reads the parameter values ​​stored in the learning result storage unit 120, and outputs the hostile person class belonging probability and the hostile action class belonging probability.

[0038] The shared adversarial classifier 1051 shown in FIG. 2 is composed of a shared processor 1052, an adversarial person classifier 1053, and an adversarial action classifier 1054. The shared processor 1052 receives action features from the action feature extractor 1031 as input, and receives person features from the person feature extractor 1041 as input. The shared processor 1052 performs common processing on the action features and person features to remove information unnecessary for action and person identification. When the shared processor 1052 receives action features from the action feature extractor 1031 as input, it inputs the processed action features to the adversarial person classifier 1053. The adversarial person classifier 1053 outputs an adversarial person class belonging probability based on the action features input from the shared processor 1052. On the other hand, when the shared processor 1052 receives person features as input from the person feature extractor 1041, it inputs the processed person features to the hostile action classifier 1054. The hostile action classifier 1054 outputs an hostile action class belonging probability based on the person features input from the shared processor 1052. The common processing performed by the shared processor 1052 on the action features and person features is part of the processing for calculating the hostile person class belonging probability of the action input data, and is also part of the processing for calculating the hostile action class belonging probability of the person input data.

[0039] Any function can be used for the action recognition unit 103, the person recognition unit 104, the adversarial recognition unit 105, and each of these components as long as it is differentiable with respect to the parameters. In this embodiment, a neural network is used.

[0040] The loss evaluation unit 106 is composed of an action class classification loss evaluation unit 1061, an adversarial person class classification loss evaluation unit 1062, an adversarial action class classification loss evaluation unit 1063, and a person class classification loss evaluation unit 1064.

[0041] The action class identification loss evaluation unit 1061 receives as input the estimated action class belonging probability for the person action data to be processed from the action identification unit 103, and further receives a given teacher action class belonging probability for the person action data to be processed. The teacher action class belonging probability represents the correct belonging probability for each action class for the person action data to be processed. The teacher action class belonging probability may be read out from the data storage unit 110, for example, together with the person action data set. The action class identification loss evaluation unit 1061 calculates the value of the action class identification loss function (hereinafter referred to as the "action class identification loss evaluation value"), which is a loss evaluation function representing the difference between the estimated action class belonging probability and the teacher action class belonging probability.

[0042] The hostile person class identification loss evaluation unit 1062 receives as input the hostile person class belonging probability for the processing target person motion data from the adversarial identification unit 105, and further receives a given teacher person class belonging probability for the processing target person motion data. The teacher person class belonging probability represents the correct belonging probability for the processing target person motion data to each person class. The teacher person class belonging probability may be read out from the data storage unit 110 together with the person motion data set, for example. The hostile person class identification loss evaluation unit 1062 calculates the value of the hostile person class identification loss function (hereinafter referred to as the "hostile person class identification loss evaluation value"), which is a loss evaluation function representing the difference between the hostile person class belonging probability and the teacher person class belonging probability, and a value indicating the ambiguity of the hostile person class belonging probability (hereinafter referred to as the "entropy value of the hostile person class belonging probability").

[0043] The adversarial action class identification loss evaluation unit 1063 receives as input the adversarial action class belonging probability for the processing target human action data from the adversarial identification unit 105, and further receives a given teacher action class belonging probability for the processing target human action data. The teacher action class belonging probability represents the correct belonging probability for each action class for the processing target human action data. The teacher action class belonging probability may be read out from the data storage unit 110 together with the human action data set, for example. The adversarial action class identification loss evaluation unit 1063 calculates the value of the adversarial action class identification loss function (hereinafter referred to as the "adversarial action class identification loss evaluation value"), which is a loss evaluation function representing the difference between the adversarial action class belonging probability and the teacher action class rule probability, and a value indicating the ambiguity of the adversarial action class belonging probability (hereinafter referred to as the "entropy value of the adversarial action class belonging probability").

[0044] The person class identification loss evaluation unit 1064 receives as input an estimated person class membership probability for the person motion data to be processed from the person identification unit 104, and further receives a given teacher person class membership probability for the person motion data to be processed. The teacher person class membership probability represents a correct attribution probability for each person class for the person motion data to be processed. The teacher person class membership probability may be read out from the data storage unit 110 together with the person motion data set, for example. The person class identification loss evaluation unit 1064 calculates a value of a person class identification loss function (hereinafter referred to as a "person class identification loss evaluation value") which is a loss evaluation function representing the difference between the estimated person class membership probability and the teacher person class membership probability.

[0045] The learning unit 107 shown in FIG. 1 receives as inputs from the loss evaluation unit 106 an action class identification loss evaluation value, an adversarial person class identification loss evaluation value, an entropy value of the adversarial person class belonging probability, an adversarial action class identification loss evaluation value, an entropy value of the adversarial action class belonging probability, and a person class identification loss evaluation value. The learning unit 107 uses these input values ​​to learn parameter values ​​used in the processing of each of the action identification unit 103, the person identification unit 104, and the adversarial identification unit 105. The learning unit 107 records the parameter values ​​used in the processing of each of the action identification unit 103, the person identification unit 104, and the adversarial identification unit 105 obtained by learning in the learning result storage unit 120. Learning regarding the parameter values ​​used in the processing of the action identification unit 103 is performed so as to reduce the action class identification loss evaluation value and to increase the entropy value of the adversarial person class identification loss evaluation value and the adversarial action class belonging probability. The parameter values ​​used in the processing of the person identification unit 104 are learned so as to reduce the person class classification loss evaluation value and to increase the hostile action class classification loss evaluation value and the entropy value of the hostile person class membership probability. The parameter values ​​used in the processing of the adversarial identification unit 105 are learned so as to reduce the hostile action class classification loss evaluation value and the hostile person class classification loss evaluation value.

[0046] <Configuration example of action recognition device> FIG. 3 is a functional block diagram showing an example of the action identification device 200 according to the present embodiment. The action identification device 200 is configured using an information processing device such as a personal computer or a server device. The action identification device 200 includes a control unit 92 and a storage unit 230. The control unit 92 is configured using a processor such as a CPU and a memory. The control unit 92 functions as the action identification unit 201 by the processor executing a program. Note that all or part of the functions of the control unit 92 may be implemented using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded in a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (e.g., SSD), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0047] The storage unit 230 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 230 stores parameter values ​​as learning results obtained by the learning performed by the learning unit 107 of the action discrimination learning device 100. The parameter values ​​are information stored in the learning result storage unit 120, for example.

[0048] When the motion identification unit 201 receives the processing target person motion data 210 (the prediction target person motion data), it reads out parameter values ​​from the storage unit 230 and operates based on the read out parameter values. The motion identification unit 201 calculates the estimated motion class belonging probability for the processing target person motion data 210. The output of the motion identification unit 201 may be the belonging probability itself for each motion class for the processing target person motion data 210, or may be information indicating the estimated result of which motion class the processing target person motion data 210 belongs to. For example, the motion identification unit 201 may read out parameter values ​​used in the processing of the motion identification unit 103 from the storage unit 230, and output the belonging probability for each motion class calculated by performing the same processing as the motion identification unit 103 on the human motion data 210 using the read out parameter values, or may output the motion class with the highest calculated belonging probability.

[0049] <Example of the motion discrimination learning device> Fig. 4 is a flowchart showing an example of the operation of the action discrimination learning device 100. An example of the operation of the action discrimination learning device 100 will be described with reference to Fig. 4. The action discrimination learning device 100 receives a set of human action data from the data storage unit 110, and executes a learning process routine shown in Fig. 4.

[0050] First, the control unit 90 of the action identification learning device 100 reads a person action data set from the data storage unit 110 (step S101). Next, the control unit 90 inputs each person action data included in the input person action data set to the action input preprocessing unit 101 and the person input preprocessing unit 102. The action input preprocessing unit 101 performs preprocessing on the person action data to generate action input data, and the person input preprocessing unit 102 performs preprocessing on the person action data to generate person input data. As a result, the control unit 90 generates two pieces of data, the action input data and the person input data (step S102).

[0051] The action identification unit 103 obtains an estimated action class belonging probability by executing a process on the action input data. The person identification unit 104 obtains an estimated person class belonging probability by executing a process on the person input data. The adversarial identification unit 105 obtains an adversarial person class belonging probability by executing a process on the action input data. Furthermore, the adversarial identification unit 105 obtains an adversarial action class belonging probability by executing a process on the person input data. As a result, the control unit 90 obtains an estimated action class belonging probability, an estimated person class belonging probability, an adversarial person class belonging probability, and an adversarial action class belonging probability (step S103).

[0052] The loss evaluation unit 106 performs processing based on the estimated action class belonging probability, the estimated person class belonging probability, the hostile action class belonging probability, and the hostile person class belonging probability obtained in step S103, to obtain an action class identification loss evaluation value, an hostile person class identification loss evaluation value, an entropy value of the hostile person class belonging probability, an hostile action class identification loss evaluation value, an entropy value of the hostile action class belonging probability, and a person class identification loss evaluation value (step S104).

[0053] After the process of step S104 is completed, the learning unit 107 updates the values ​​of parameters used in the processes of the action recognition unit 103, the person recognition unit 104, and the adversarial recognition unit 105 based on the values ​​obtained in the process of step S104 (step S105). The learning unit 107 records the updated parameter values ​​in the learning result storage unit 120.

[0054] Then, the control unit 90 judges whether or not the termination condition is satisfied (step S106). If the control unit 90 judges that the termination condition is not satisfied (step S106-NO), the control unit 90 returns to step S101 and repeats the process. On the other hand, if the control unit 90 judges that the termination condition is satisfied (step S106-YES), the control unit 90 ends the process.

[0055] By the iterative learning described above, parameter values ​​of the action recognition unit 103, the person recognition unit 104, and the adversarial recognition unit 105 are learned. The learning of the parameters used in the processing of the action recognition unit 103 is performed so as to reduce the action class recognition loss evaluation value and increase the entropy value of the hostile person class recognition loss evaluation value and the hostile action class belonging probability. The learning of the parameters used in the processing of the person recognition unit 104 is performed so as to reduce the person class recognition loss evaluation value and increase the entropy value of the hostile action class recognition loss evaluation value and the hostile person class belonging probability. The output of the parameters used in the processing of the adversarial recognition unit 105 is performed so as to reduce the hostile action class recognition loss evaluation value and the hostile person class recognition loss evaluation value.

[0056] <Details of each process> [Processing in the motion input pre-processing section and the person input pre-processing section] Next, the processing of each processing unit of action discrimination learning device 100 will be described in detail. First, the process of the action input pre-processing unit 101 and the process of the person input pre-processing unit 102 in step S102 will be described in detail.

[0057] The action recognition unit 103 is required to perform action recognition by focusing on the action features of the person action data, and the person recognition unit 104 is required to perform person recognition by focusing on the person features of the person action data. To achieve each objective, features that are not the focus of each become obstacles. Therefore, in this embodiment, this obstacle is mitigated by introducing a preprocessing that removes as many person features as possible from the action input data, and a preprocessing that removes as many action features as possible from the person input data.

[0058] There are various types of pre-processing for motion input and pre-processing for human input that can satisfy these requirements. A specific example of these pre-processing processes will be given below for the case where human motion data is given in the form of human joint point time-series data.

[0059] First, the motion input preprocessing performed by the motion input preprocessing unit 101 may involve an operation of randomly expanding or contracting the length between joint points. This operation causes the loss of physical characteristics such as the person's height and proportions, while preserving motion characteristics such as joint angles and time-series displacement. Therefore, this operation meets the requirements for motion input preprocessing.

[0060] On the other hand, the person input pre-processing performed by the person input pre-processing unit 102 may be a process of randomly changing the time series order of the action frames of the person action data. Although this operation causes the loss of time series information of the action (information for identifying the action), the physical characteristics of the subject of the action can be preserved by changing the order. Therefore, this operation satisfies the requirements for person input pre-processing.

[0061] In addition to these, there are various other forms of preprocessing. Any preprocessing can be used as long as it satisfies the above requirements. In the following, we will refer to the family of motion input preprocessing that satisfies the requirements as T A Let T be a family of person input preprocessing that satisfies the requirements. S When applying the human motion data x to the real world, we randomly select a motion input preprocessing T A , Person Input Preprocessing T S Select and apply, and enter the operation data x A =T A (x), person input data x S =T S State that (x) is obtained.

[0062] [Processing of the Action Recognition Unit, Person Recognition Unit, and Adversarial Recognition Unit] The processing in step S103 of the action recognition unit 103, the person recognition unit 104, and the adversarial recognition unit 105 will be described. The action feature extractor 1031, which is a common part of the action recognition unit 103 and the adversarial recognition unit 105, extracts a parameter θ A and the motion input data x generated in step S102 A The input is a vector z A A function F that outputs AThe person feature extractor 1041, which is a common part between the person identification unit 104 and the adversarial identification unit 105, uses the parameter θ S and the person input data x generated in step S102 S The input is a vector z S A function F that outputs S This is achieved by using

[0063] The action classifier 1032 of the action classifier 103 uses a parameter φ A and the motion feature vector z A The estimated action class membership probability can be expressed as a function F A Using this, it can be expressed as a probability function as shown in the following equation (1).

[0064]

number

[0065] Equation (1) is based on the parameter θ A , parameter φ A , and motion input data x A Given the probability of belonging to each action class, y A is the probability of occurring.

[0066] The person classifier 1042 of the person identification unit 104 uses a parameter φ S and the person feature vector z S The estimated person class membership probability can be expressed as a function that receives as input the function F S Using this, it can be expressed as a probability function as shown in the following equation (2).

[0067]

number

[0068] Equation (2) is based on the parameter θ S , parameter φ S , and person input data x S Given the probability of belonging to each person class, y S is the probability of occurring.

[0069] The shared adversarial classifier 1051 of the adversarial classifier 105 has a parameter φ Adv share and parameter φ Adv S and the motion feature vector z A The adversarial class membership probability can be expressed as a function that outputs the adversarial class membership probability when it receives the input. The adversarial class membership probability can be expressed as a function F A Using this, it can be expressed as a probability function as shown in the following equation (3).

[0070]

number

[0071] Equation (3) is the parameter θ A , parameter φ Adv share , parameter φ Adv S , and motion input data x A Given the probability of belonging to each person class, y Adv S is the probability of occurring.

[0072] In addition, the shared adversarial classifier 1051 of the adversarial classifier 105 has a parameter φ Adv share and parameter φ Adv A and the person feature vector z S The adversarial action class membership probability can be expressed as a function that outputs the adversarial action class membership probability when it receives the input. The adversarial action class membership probability can be expressed as a function F SUsing this, it can be expressed as a probability function as shown in the following equation (4).

[0073]

number

[0074] Equation (4) is based on the parameter θ S , parameter φ Adv share , parameter φ Adv A , and person input data x S Given the probability of belonging to each action class, y Adv A is the probability of occurring.

[0075] The shared adversarial classifier 1051 may be composed of three parts: a shared processor 1052, an adversarial person classifier 1053, and an adversarial action classifier 1054. In this case, the shared processor 1052 is configured to use a parameter φ Adv share The adversarial human classifier 1053 has parameters φ Adv S The adversarial action classifier 1054 has parameters φ Adv A By appropriately designing the shared adversarial classifier 1051 as a function having the above-mentioned structure, it is possible to configure the shared adversarial classifier 1051 capable of the above processing.

[0076] [Processing of the action class identification loss evaluation unit and the person class identification loss evaluation unit] The processing performed by the action class classification loss evaluation unit 1061 and the person class classification loss evaluation unit 1064 of the loss evaluation unit 106 in step S104 will be described.

[0077] First, a method for calculating a motion class classification evaluation value will be described. The desirable motion classification unit 103 calculates human motion data x G Operation input t A Operation input data x applied G A , and the teacher action class membership probability yG A In other words, the action classifier 103 obtains the probability of identification of the correct action class. G A and the corresponding teacher action class membership probability y G A The probability of occurrence of p(x G A ,y G A ), then in learning, the parameter θ A , φ A It is sufficient to be able to determine the above.

[0078]

number

[0079] Here, E(·) is the expected value. In this embodiment, the training data set is given in the form of a set of pairs of human action data and teacher action class membership probability. Therefore, the expected value is approximately replaced by the sum as shown in the following formula (6).

[0080]

number

[0081] where X is one or more person motion data, and Y is A is the set of teacher action class membership probabilities corresponding to X. Also, x G T applied to A is the family of motion input preprocessing T A For each data input, the loss function is randomly selected from the following: A The value evaluated against is the operational class discrimination loss evaluation value.

[0082] Equation (6) is expressed as θ A , φ A By making it smaller, we can make Y AIt is possible to obtain a desirable motion recognition unit 103 that can output such a θ A , φ A It is known that local minimization is possible when the function is differentiable with respect to . Therefore, in one example of this embodiment, functions that satisfy the following conditions may be selected as the motion feature extractor 1031 and the motion discriminator 1032 of the motion discrimination unit 103.

[0083] The conditions for the motion feature extractor 1031 are as follows. (1) Operation input data X A When input, the motion feature vector z A It must be a function that outputs (2)θ A It is differentiable with respect to

[0084] The conditions for the action identifier 1032 are as follows: (1) Motion feature vector z A The estimated action class discrimination probability y A It must be a function that outputs (2)φ A It is differentiable with respect to

[0085] Next, a method for calculating a person class identification evaluation value will be described. In this embodiment, person identification is also learned in parallel with action identification. The desired person identification unit 104 obtains person action data x G Enter the person in S Applying the human input data x G S , and the teacher class membership probability y G S In other words, the person identification unit 104 is capable of calculating the probability of identifying the correct person class. G S and the corresponding teacher class membership probability y G S The probability of occurrence of p(x G S ,y G S), then in learning, the parameter θ S , φ S It is sufficient to be able to determine the above.

[0086]

number

[0087] In this embodiment, the training data set is given in the form of a set of pairs of person action data and teacher person class membership probability. Therefore, the expected value is approximately replaced by the sum as shown in the following formula (8).

[0088]

number

[0089] where X is one or more person motion data, and Y is S is the set of teacher class membership probabilities corresponding to X. Also, x G T applied to S is the person input preprocessing family T S For each data input, the loss function is randomly selected from the following: S The value evaluated for is the person class identification loss evaluation value.

[0090] Equation (8) is expressed as θ S , φ S By making it smaller, we can make Y S It is possible to obtain a desirable person identification unit 104 that can output such a θ S , φ S It is known that local minimization is possible when the function is differentiable with respect to . Therefore, in one example of this embodiment, a function that satisfies the following condition may be selected for the person feature extractor 1041 and the person classifier 1042 of the person identification unit 104.

[0091] The conditions for the person feature extractor 1041 are as follows. (1) Person input data X S The person feature vector z of the person input data is input. S It must be a function that outputs (2)θ S It is differentiable with respect to

[0092] The conditions for the person identifier 1042 are as follows. (1) Person feature vector z S The estimated person class identification probability y S It must be a function that outputs (2)φ S It is differentiable with respect to

[0093] [Processing of the adversarial person class identification loss evaluation unit and the adversarial action class identification loss evaluation unit] The processing performed in step S104 by the hostile person class classification loss evaluation unit 1062 and the hostile action class classification loss evaluation unit 1063 of the loss evaluation unit 106 will be described.

[0094] The shared adversarial classifier 1051 of the adversarial classifier 105 extracts the motion feature vector z A As an input, the shared adversarial classifier 1051 estimates the person class and outputs the adversarial person class membership probability. The desired behavior of such a shared adversarial classifier 1051 is to estimate the person action data x G Input preprocessing A Operation input data x applied G A is input to the motion feature extractor 1031, and the motion input data x G A The motion feature vector z G A When the input is received, the teacher person class belonging probability y G S The purpose is to output the operation input data x G Aに The corresponding teacher class membership probability y G S The probability of occurrence of p(xG A ,y G S ), then in the training of the shared adversarial classifier 1051, the parameter φ is set so that the value of equation (9) becomes small. Adv share , φ Adv S It is sufficient to be able to determine the above.

[0095]

number

[0096] In this embodiment, the training data set is given in the form of a set of pairs of person action data and teacher person class membership probability. Therefore, the expected value is approximately replaced by the sum as shown in the following formula (10).

[0097]

number

[0098] where X is one or more person motion data, and Y is S is the set of teacher class membership probabilities corresponding to X. Also, x G T applied to S is the person input preprocessing family T S For each data input, the adversarial person class identification loss evaluation value in this embodiment is selected randomly from X, Y. S The value evaluated against is the adversarial human class identification loss evaluation value.

[0099] The adversarial class classification loss evaluation unit 1062 evaluates the entropy value H S (y Adv S |x A The entropy value of the hostile class membership probability is calculated as shown in the following equation (11).

[0100]

number

[0101] However, K S is a constant representing the total number of person classes. The entropy value of the hostile person class membership probability shown in equation (11) is a value indicating the ambiguity of the hostile person class membership probability, which is small when the hostile person class membership probability indicates a value for one person class and is largest when the hostile person class has a similar value.

[0102] In addition, the shared adversarial classifier 1051 of the adversarial classifier 105 extracts the person feature vector z S As an input, the shared adversarial classifier 1051 estimates the action class and outputs the adversarial action class membership probability. The desired behavior of such a shared adversarial classifier 1051 is to estimate the human action data x G Pre-processing of human input S Applying the human input data x G S is input to the person feature extractor 1041, and the person input data x G S The person feature vector z G S When the input is received, the teacher action class belonging probability y G A The output of the person input data x G S and the corresponding teacher action class membership probability y G A The probability of occurrence of p(x G S ,y G A ), then in the training of the shared adversarial classifier 1051, the parameter φ is set so that the value of equation (12) becomes small. Adv share , φ Adv A It is sufficient to be able to determine the above.

[0103]

number

[0104] In this embodiment, the training data set is given in the form of a set of pairs of human motion data and teacher motion class membership probability. Therefore, the expected value is approximately replaced by the sum as shown in the following formula (13).

[0105]

number

[0106] where X is one or more person motion data, and Y is A is the set of teacher action class membership probabilities corresponding to X. Also, x G T applied to A is the person input preprocessing family T A For each data input, the adversarial action class identification loss evaluation value in this embodiment is selected randomly from X, Y. A The value evaluated against is the adversarial action class identification loss evaluation value.

[0107] The adversarial action class classification loss evaluation unit 1063 evaluates the entropy value H A (y Adv A |x S The entropy value of the adversarial action class membership probability is calculated as shown in the following equation (14).

[0108]

number

[0109] However, K A is a constant representing the total number of action classes.

[0110] Regarding equation (10), φ Adv share , φ Adv S By reducing , the shared adversarial classifier 1051 can learn to classify people from motion features, and at the same time, φ Adv share , φ Adv A By reducing φ, the shared adversarial classifier 1051 can learn to classify people from motion features. The learning of the shared adversarial classifier 1051 of the adversarial classifier 105 is performed by Adv share , φ Adv A , φ Adv S It is known that if equations (10) and (13) are differentiable, local minimization is possible. Therefore, in one example of this embodiment, a function that satisfies the following condition may be selected as the shared adversarial classifier 1051 of the adversarial classifier unit 105.

[0111] The conditions for the shared adversarial classifier 1051 are as follows: (1) Person feature vector z S With the input, the adversarial action class membership probability y Adv A and outputs the motion feature vector z A With the input, the adversarial class membership probability y Adv S This is a function that outputs (2)φ Adv share , φ Adv A , φ Adv S It is differentiable with respect to

[0112] [Learning process] The process of the learning unit 107 in step S105 will be described. The outline of the processing realized by the learning unit 107 is as follows. First, the action feature extractor 1031 and the action classifier 1032 of the action classification unit 103 are expected to correctly classify actions without being affected by the person information of the person action data, and the person feature extractor 1041 and the person classifier 1042 of the person classification unit 104 are expected to correctly classify people without being affected by the action information of the person action data. The action input preprocessing unit 101 and the person input preprocessing unit 102 apply preprocessing that loses as much unnecessary information as possible for each input, but it is difficult to design preprocessing that completely loses information. Therefore, the learning unit 107 in this embodiment has the action feature extractor 1031 and the person feature extractor 1041 as common parts and is provided with a shared adversarial classifier 1051. By performing adversarial learning using the shared adversarial classifier 1051, a learning process is executed in which information that is not necessary for classification is removed from the motion features output by the motion feature extractor 1031 and the person features output by the person feature extractor 1041.

[0113] Here, the effect of providing a shared adversarial classifier 1051 that shares some parameters, rather than providing an individual adversarial classifier for each classification, will be described. In the shared adversarial classifier 1051, both action classification and person classification are learned in an adversarial manner. Therefore, in the shared adversarial classifier 1051, features that are common to the two adversarial classification learnings are learned. This makes it possible to capture common information that is unnecessary for both action classification and person classification (for example, posture information when a certain person performs a certain action is neither information representing a certain person nor information representing a certain action) more efficiently than in the case of learning individual adversarial classifiers.

[0114] Now, the learning process of the learning unit 107 for each processor will be described. First, the learning unit 107 learns parameters of the action classifier 1032 of the action recognition unit 103 so as to minimize an action class recognition loss evaluation value, and learns parameters of the person classifier 1042 of the person recognition unit 104 so as to minimize a person class recognition loss evaluation value. Specifically, the learning unit 107 learns so as to optimize the problems shown in equations (15) and (16).

[0115]

number

[0116]

number

[0117] Next, the learning unit 107 learns parameters of the action feature extractor 1031 of the action recognition unit 103 so as to minimize the action class recognition loss evaluation value, maximize the entropy value of the hostile action class membership probability, and maximize the hostile action class recognition loss evaluation value. Also, the learning unit 107 learns parameters of the person feature extractor 1041 of the person recognition unit 104 so as to minimize the person loss evaluation value, maximize the entropy value of the hostile action class membership probability, and maximize the hostile action class recognition loss evaluation value. Specifically, the learning unit 107 learns so as to optimize the problems shown in Equations (17) and (18).

[0118]

number

[0119]

number

[0120] where λ Adv S , λ Adv A , λ E-Adv S , λ E-Adv A is a parameter for balancing the learning of each of the learnings. This value may be a constant that is independent of the number of learning iterations, or may be a value that changes according to a predetermined schedule depending on the number of learning iterations.

[0121] By performing action identification learning as in equation (17), action feature extractor 1031 can extract action features from action input data, and by maximizing the entropy value of the adversarial learning against people and the estimated person class membership probability, action feature extractor 1031 is trained so that the extracted action features do not include person features. Similarly, person feature extractor 1041 is trained by equation (18) so that it can extract person features that do not include action features.

[0122] Finally, the learning unit 107 learns parameters of the shared adversarial classifier 1051 of the adversarial classifier 105 so as to minimize the adversarial action class classification loss evaluation value and the adversarial person class classification loss evaluation value. Specifically, the learning unit 107 learns so as to optimize the problem shown in equation (19).

[0123]

number

[0124] Here, each learning is performed using the action class discrimination loss evaluation value L A , person class identification loss evaluation value L S , adversarial action class identification loss evaluation value L Adv A , adversarial class classification loss evaluation value L Adv S , the entropy value of the adversarial action class membership probability H A , the entropy value of the adversarial class membership probability H S is the parameter θ A , θ S , φ A , φ S , φ Adv share , φ Adv A , φ Adv S To choose the function for each processor to be differentiable with respect to , it is possible to train it by error gradient descent.

[0125] [Parameter storage process] After the parameter learning, in step S105, the learning unit 107 learns the parameter θ A , θ S , φ A , φ S , φ Adv share , φ Adv A , φ Adv S is stored in the learning result storage unit 120.

[0126] The learning process from step S101 to step S105 described above may be repeated until a termination condition is satisfied. Any information may be used as the termination condition. For example, the termination condition may be that a predetermined number of repetitions are performed, that the value of the objective function does not change by a certain amount, or that the accuracy of evaluation data prepared separately from the training data does not change by a certain amount.

[0127] <Operation of the motion recognition device> An example of the operation of action identification device 200 will be described. First, the control unit 92 of the action recognition device 200 reads person action data 210. Next, the action recognition unit 201 of the control unit 92 reads the learning result parameters recorded by the action recognition learning device 100 from the storage unit 230. Having read the parameters, the action recognition unit 201 performs the same processing as the action recognition unit 103 on the person action data 210 to obtain a recognition result 220.

[0128] Fig. 5 is a schematic diagram showing a hardware configuration common to each device. Each of the action identification learning device 100 and the action identification device 200 may be configured using an information processing device 900 as shown in Fig. 5. The information processing device 900 includes a processor 901, a memory 902, and an auxiliary storage device 903. The processor 901, the memory 902, and the auxiliary storage device 903 are communicatively connected via a bus 904.

[0129] In this embodiment configured as above, the problem of the existence of missing action classes for each person is addressed by performing learning to separate action-dependent features and person-dependent features of the data.

[0130] The data storage unit 110 may be provided in the action identification learning device 100. The learning result storage unit 120 may be provided outside the action identification learning device 100. The storage unit 230 may be provided outside the action identification device 200. When each storage unit is provided outside the respective device, the action data may be acquired by performing communication using, for example, TCP / IP (Transmission Control Protocol / Internet Protocol) or the like.

[0131] Each of the action identification learning device 100 and the action identification device 200 may be implemented using one information processing device, or may be distributed and implemented in multiple information processing devices. When the action identification learning device 100 and the action identification device 200 are distributed and implemented in multiple information processing devices, it is possible to arbitrarily determine which functional unit is realized by which information processing device. Furthermore, the same functional unit may be realized by multiple information processing devices.

[0132] According to the above-described embodiment, the action identification learning device includes an action input preprocessing unit, a person input preprocessing unit, an action identification unit, a person identification unit, an adversarial identification unit, a loss evaluation unit, and a learning unit. The action input preprocessing unit generates action input data representing characteristics related to action from person action data, which is data representing a person's action. The person input preprocessing unit generates person input data representing characteristics related to a person from the person action data. The action identification unit calculates an estimated action class belonging probability, which is an estimated probability that the action input data belongs to each of one or more action classes. The person identification unit calculates an estimated person class belonging probability, which is an estimated probability that the person input data belongs to each of one or more person classes. The adversarial identification unit calculates an adversarial person class belonging probability, which is an estimated probability that the action input data belongs to each of one or more person classes, and an adversarial action class belonging probability, which is an estimated probability that the person input data belongs to a action class. The loss evaluation unit calculates a loss evaluation value representing an estimation error based on the estimated action class belonging probability, the estimated person class belonging probability, the hostile action class belonging probability, the hostile person class belonging probability, a given teacher action class belonging probability indicating the probability that the person action data belongs to each of the action classes, and a given teacher person class belonging probability indicating the probability that the person action data belongs to each of the person classes. The learning unit learns, based on the loss evaluation value, a first parameter value used by the action identification unit in the generation process of the action input data, a second parameter value used by the person identification unit in the generation process of the person input data, a third parameter value used by the hostile identification unit in the calculation process of the hostile person class belonging probability, and a fourth parameter value used by the hostile identification unit in the calculation process of the hostile action class belonging probability.

[0133] The first parameter, the second parameter, the third parameter, and the fourth parameter may be multiple. When the third parameter and the fourth parameter are multiple, some of the third parameter and the fourth parameter may be the same. For example, the first parameter is θ A and φ A and the second parameter corresponds to θ S and φ S and the third parameter corresponds to φAdv share and φ Adv S The fourth parameter corresponds to φ Adv share and φ Adv A is equivalent to.

[0134] The motion input pre-processing unit may generate the motion input data by performing a process on the person motion data to cause information identifying a person to be lost. The person input pre-processing unit may generate the person input data by performing a process on the person motion data to cause information identifying a motion to be lost.

[0135] The action identification unit may include a action feature extractor that extracts action features from the action input data, and an action classifier that outputs an estimated action class belonging probability from the action feature. The person identification unit may include a person feature extractor that extracts person features from the action input data, and a person classifier that outputs an estimated person class belonging probability from the person feature. The adversarial identification unit calculates an adversarial person class belonging probability based on the action features extracted by the action feature extractor, and calculates an adversarial action class belonging probability based on the person features extracted by the person feature extractor.

[0136] The adversarial identification unit may include a shared processor, a hostile person classifier, and a hostile action classifier. The shared processor performs processing on the action features and the person features to remove information unnecessary for action identification and person identification. The hostile person classifier calculates an adversarial person classifier membership probability based on the action features processed by the shared processor. The adversarial action classifier calculates an adversarial action classifier membership probability based on the person features processed by the shared processor.

[0137] The loss evaluation unit calculates an action class identification loss evaluation value that becomes smaller as the similarity between the estimated action class belonging probability and the teacher action class belonging probability becomes higher, a person class identification loss evaluation value that becomes smaller as the similarity between the estimated person class belonging probability and the teacher person class belonging probability becomes higher, an adversarial action class identification loss evaluation value that becomes smaller as the similarity between the adversarial action class belonging probability and the teacher action class belonging probability becomes higher, an adversarial character class identification loss evaluation value that becomes smaller as the similarity between the adversarial character class belonging probability and the teacher person class belonging probability becomes higher, a first entropy value indicating the ambiguity of the adversarial action class belonging probability, and a second entropy value indicating the ambiguity of the adversarial character class belonging probability. For example, the first entropy corresponds to the entropy value of the adversarial action class belonging probability in the embodiment, and the second entropy corresponds to the entropy value of the adversarial character class belonging probability in the embodiment. The learning unit learns the value of a first parameter so as to reduce the action class identification loss evaluation value and increase the hostile person class identification loss evaluation value and the first entropy value, learns the value of a second parameter so as to reduce the person class identification loss evaluation value and increase the hostile action class identification loss evaluation value and the second entropy value, and learns the values ​​of a third parameter and a fourth parameter so as to reduce the hostile action class identification loss evaluation value and the hostile person class identification loss evaluation value.

[0138] The motion identification device includes a motion identification unit that obtains a probability of belonging of the human motion data to a motion class based on a value of a first parameter obtained by the motion identification learning device.

[0139] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0140] 100...action identification learning device, 101...action input preprocessing unit, 102...person input preprocessing unit, 103...action identification unit, 104...person identification unit, 105...adversarial identification unit, 106...loss evaluation unit, 107...learning unit, 110...data storage unit, 120...learning result storage unit, 90...control unit, 1031...action feature extractor, 1032...action classifier, 1041...person feature extractor, 1042...person classifier, 1051...shared adversarial classifier, 1052...shared processor, 1053...adversarial person Classifier, 1054...adversarial action classifier, 1061...action class classification loss evaluation unit, 1062...adversarial person class classification loss evaluation unit, 1063...adversarial action class classification loss evaluation unit, 1064...person class classification loss evaluation unit, 200...action classification device, 201...action classification unit, 210...person action data, 220...classification result, 230...storage unit, 92...control unit, 900...information processing device, 901...processor, 902...memory, 903...auxiliary storage device, 904...bus

Claims

1. a motion input preprocessing unit that generates motion input data representing motion-related features from human motion data that is data representing a human motion; a person input pre-processing unit that generates person input data representing characteristics related to a person from the person motion data; a motion identification unit that calculates an estimated motion class membership probability, which is an estimated probability that the motion input data belongs to each of one or more motion classes; a person identification unit that calculates an estimated person class membership probability, which is an estimated probability that the person input data belongs to each of one or more person classes; an adversarial identification unit that calculates an adversarial person class membership probability, which is an estimated probability that the action input data belongs to each of one or more person classes, and an adversarial action class membership probability, which is an estimated probability that the person input data belongs to an action class; a loss evaluation unit that calculates a loss evaluation value that represents an error of estimation based on the estimated action class belonging probability, the estimated person class belonging probability, the hostile action class belonging probability, the hostile person class belonging probability, a given teacher action class belonging probability indicating a probability that the person action data belongs to each of the action classes, and a given teacher person class belonging probability indicating a probability that the person action data belongs to each of the person classes; a learning unit that learns, based on the loss evaluation value, a value of a first parameter used by the action identification unit in a process of generating the action input data, a value of a second parameter used by the person identification unit in a process of generating the person input data, a value of a third parameter used by the adversarial identification unit in a process of calculating the hostile person class membership probability, and a value of a fourth parameter used by the adversarial identification unit in a process of calculating the hostile action class membership probability; A motion discrimination learning device comprising:

2. the motion input preprocessing unit generates the motion input data by performing a process on the person motion data to eliminate information that identifies a person; and the person input pre-processing unit generates the person input data by performing a process on the person motion data to eliminate information identifying motions. The motion discrimination learning apparatus according to claim 1 .

3. The action identification unit is a motion feature extractor that extracts a motion feature from the motion input data; a motion classifier that outputs the estimated motion class membership probability from the motion feature amount, The person identification unit includes: a person feature extractor that extracts person features from the action input data; a person classifier that outputs the estimated person class membership probability from the person feature amount, The adversarial identification unit calculates the hostile person class belonging probability based on the action feature amount extracted by the action feature extractor, and calculates the adversarial action class belonging probability based on the person feature amount extracted by the person feature extractor. The motion discrimination learning apparatus according to claim 1 .

4. The adversarial classifier comprises: a shared processor that performs a process for removing information unnecessary for identifying a motion and a person from the motion feature amount and the person feature amount; an adversarial person classifier that calculates an adversarial person class belonging probability based on the motion feature amount processed by the shared processor; and an adversarial action classifier that calculates an adversarial action class belonging probability based on the person feature amount processed by the shared processor. The motion discrimination learning apparatus according to claim 3 .

5. The loss evaluation unit includes: an action class identification loss evaluation value that becomes smaller as the similarity between the estimated action class membership probability and the teacher action class membership probability becomes higher; a person class identification loss evaluation value that becomes smaller as the similarity between the estimated person class membership probability and the teacher person class membership probability becomes higher; an adversarial action class identification loss evaluation value that becomes smaller as the similarity between the adversarial action class membership probability and the teacher action class membership probability becomes higher; an adversarial character class identification loss evaluation value that becomes smaller as the similarity between the adversarial character class membership probability and the teacher character class membership probability becomes higher; a first entropy value indicating the ambiguity of the adversarial action class membership probability; A second entropy value indicating the ambiguity of the hostile person class membership probability is calculated; The learning unit is learning values ​​of the first parameter so as to reduce the action class discrimination loss evaluation value and increase the adversarial person class discrimination loss evaluation value and the first entropy value; learning a value of the second parameter so as to reduce the person class identification loss evaluation value and increase the adversarial action class identification loss evaluation value and the second entropy value; learning values ​​of the third parameter and the fourth parameter so as to reduce the adversarial action class identification loss evaluation value and the adversarial person class identification loss evaluation value; The motion discrimination learning apparatus according to claim 1 .

6. a motion identification unit that calculates a probability of belonging of human motion data to a motion class by using a value of the first parameter learned in the motion identification learning device according to any one of claims 1 to 5; A motion identification device comprising:

7. a motion input pre-processing step of generating motion input data representing motion-related features from human motion data which is data representing a human motion; a person input pre-processing step of generating person input data representing characteristics related to a person from the person motion data; a motion identification step of calculating an estimated motion class membership probability, which is an estimated probability that the motion input data belongs to each of one or more motion classes; a person identification step of calculating an estimated person class membership probability, which is an estimated probability that the person input data belongs to each of one or more person classes; an adversarial identification step of calculating an adversarial person class membership probability, which is an estimated probability that the action input data belongs to each of one or more person classes, and an adversarial action class membership probability, which is an estimated probability that the person input data belongs to an action class; a loss evaluation step of calculating a loss evaluation value representing an estimation error based on the estimated action class belonging probability, the estimated person class belonging probability, the adversarial action class belonging probability, the adversarial person class belonging probability, a given teacher action class belonging probability indicating a probability that the person action data belongs to each of the action classes, and a given teacher person class belonging probability indicating a probability that the person action data belongs to each of the person classes; a learning step of learning, based on the loss evaluation value, a value of a first parameter used in a process of generating the action input data in the action identification step, a value of a second parameter used in a process of generating the person input data in the person identification step, a value of a third parameter used in a process of calculating the hostile person class membership probability in the adversarial identification step, and a value of a fourth parameter used in a process of calculating the hostile action class membership probability in the adversarial identification step; A learning method that has

8. Computer, A program for causing the action discrimination learning device according to any one of claims 1 to 5 to function.

Images