Person feature extractor learning device, person prediction device, learning method, action identification learning device, action identification device, program, and information system

Abstract

To learn a motion recognition model which generalizes a variety of people even when an ideal data set for realizing person motion recognition cannot be obtained.SOLUTION: A person feature extractor learning device includes: a preprocessing part which generates preprocessed motion data by performing processing to lose information identifying a motion to motion data when the motion data is received as an input; a person feature extractor extracting a feature quantity from the motion data and the preprocessed motion data; a comparison loss evaluation part which outputs comparison loss evaluation value that is smaller the higher the similarity of a set of feature quantities is when a set of feature quantity is inputted and the set of feature quantities is extracted from the identical motion data, and is smaller the smaller the similarity of a set of feature quantities is when the set of feature quantities is extracted from motion data whose set of feature quantities mutually differs; and a learning part which learns a parameter of the person feature extractor so that the comparison loss evaluation value is small.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a person feature extractor learning device, a person prediction device, a learning method, an action recognition learning device, an action recognition device, and a program. [Background technology]

[0002] In recent years, with the spread of sensor equipment such as motion capture devices and wearable devices, and the improvement of image recognition technology, it has become easier to sense human movements. Examples of motion data obtained by sensing include data recorded by acceleration sensors installed in wearable devices and smartphones, and posture data of people in camera images. Using this data, human movements can be recognized. This type of recognition processing can be applied to various services such as person authentication, life log applications, and automatic surveillance systems.

[0003] Recognition of human actions using sensed data is often achieved by a predictive model using machine learning. A problem with this is that the accuracy of action recognition for data of people other than the person used for learning is low. This is because each person's actions are unique. For example, when considering the action of walking, there is diversity in the walking speed and the magnitude of arm swing, etc., and a predictive model trained on a different person will have a low accuracy of action recognition.

[0004] In order to solve such problems, various techniques have been disclosed. For example, Non-Patent Document 1 discloses a method for training a prediction model using data of a target person. Patent Document 1 also discloses a method for detecting a person similar to a target person from among a large number of accumulated motion data of people, and training a prediction model using the motion data of that person. Furthermore, Non-Patent Document 2 and Non-Patent Document 3 disclose a method for adapting a prediction model to people other than those included in the learning examples using active learning, transfer learning, etc. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-248017 A [Non-patent literature]

[0006] [Non-Patent Document 1] Gary Mitchell Weiss and Jeffrey Lockhart. 2012. The impact of personalization on smartphone-based activity recognition. In Workshopsat the Twenty-Sixth AAAI Conference on Artificial Intelligence. [Non-Patent Document 2] R. Fallahzadeh and H. Ghasemzadeh. 2017. Personalization without User Interruption: Boosting Activity Recognition in New Subjects Using Unlabeled Data. In 2017 ACM / IEEE 8th International Conference on Cyber-Physical Systems (ICCPS). 293-302. [Non-Patent Document 3] Seyed Ali Rokni, Marjan Nourollahi, and Hassan Ghasemzadeh. 2018. Personalized Human Activity Recognition Using Convolutional Neural Networks. In Thirty-Second AAAI Conference on Artificial Intelligence. Summary of the Invention [Problem to be solved by the invention]

[0007] Although various techniques have been proposed to achieve highly accurate action recognition, these techniques have the problem that the cost of creating training datasets to utilize them is very high.

[0008] First, in the methods described in Non-Patent Documents 1, 2, and 3, data must be acquired from a target person, and each piece of data must be given a label indicating which action it corresponds to. In addition, a model trained on a specific person does not generalize to other people, so when the target person changes, data must be acquired and labeled again.

[0009] In addition, the method described in Patent Document 1, which uses data of similar people from among data on the movements of many people, requires obtaining data and labels for all movements that need to be recognized for each person, which is very costly.

[0010] Furthermore, when creating training data, there are cases where it is not possible to obtain a person label indicating which person each action data belongs to due to privacy issues, etc. In such cases, there is a problem that all of the above technologies cannot be used because they are based on the premise that people can be identified.

[0011] As outlined above, the reality is that no technology has been proposed that can achieve human action recognition while keeping training data creation costs down under realistic conditions that take into account privacy issues, etc.

[0012] In consideration of the above circumstances, the present invention aims to provide a technology that makes it possible to learn an action recognition model that generalizes to a variety of people, even when an ideal data set for realizing human action recognition cannot be obtained. [Means for solving the problem]

[0013] One aspect of the present invention is a person feature extractor learning device that includes: a pre-processing unit that receives as input motion data, which is data that records a time series of motions, and generates pre-processed motion data by processing the motion data to lose information that identifies the motion; a person feature extractor that extracts first features from the motion data and the pre-processed motion data; a contrast loss evaluation unit that, when a set of the first features is input, outputs a contrast loss evaluation value that is smaller the higher the similarity of the set of first features is if the set of first features are extracted from the same motion data, and that is smaller the lower the similarity of the set of first features is if the set of first features are extracted from different motion data; and a learning unit that learns parameters of the person feature extractor so that the contrast loss evaluation value is smaller.

[0014] One aspect of the present invention is a person prediction device that includes a person feature extractor that receives motion data as input and outputs first features of the motion data based on parameters obtained by the above-mentioned person feature extractor learning device, and a person predictor that predicts a person in the motion data based on the first features.

[0015] One aspect of the present invention is a learning method including: a preprocessing step of receiving, as input, motion data that is data recording a time series of motions, and generating preprocessed motion data by processing the motion data to lose information that identifies the motion; a person feature extraction step of extracting first features by a person feature extractor based on the motion data and the preprocessed motion data; an evaluation step of outputting, when a set of the first features is input, a contrast loss evaluation value that becomes smaller as the similarity of the set of first features increases if the set of first features are extracted from the same motion data, and that becomes smaller as the similarity of the set of first features decreases if the set of first features are extracted from different motion data; and a learning step of learning parameters of the person feature extractor so that the contrast loss evaluation value becomes smaller.

[0016] One aspect of the present invention includes a motion feature extractor that receives as input motion data, which is data that records a time series of motions, and extracts a second feature from the motion data; a motion class classifier that calculates an estimated motion class membership probability indicating an estimated probability of membership to one or more motion classes based on the second feature; a person class classifier that calculates an estimated person class membership probability indicating an estimated probability of membership to one or more person classes based on the second feature; a motion class classification loss evaluation unit that calculates a motion class classification loss evaluation value that becomes smaller the higher the similarity between the estimated motion class membership probability and a given teacher motion class membership probability; and the estimated person class membership probability. The action classification learning device includes: a person class classification loss evaluation unit that calculates a person class classification loss evaluation value that becomes smaller as a similarity to a teacher person class membership probability based on a person prediction result obtained by the person prediction device increases; and a learning unit that learns each parameter of the action feature extractor, the action class classifier, and the person class classifier so that the action feature extractor reduces the action class classification loss evaluation value and increases the person class classification loss evaluation value, the action class classifier reduces the action class classification loss evaluation value, and the person class classifier reduces the person class classification loss evaluation value.

[0017] One aspect of the present invention is an action identification device comprising: a action feature extractor that outputs second features of input data based on parameters obtained by the above-mentioned action identification learning device; and an action class classifier that obtains, based on the parameters obtained by the action identification learning device and the second features, a probability that the data belongs to known action classes and unknown action classes.

[0018] One aspect of the present invention includes a feature extraction step in which, upon receiving as input, motion data, which is data recording a time series of motions, an motion feature extractor outputs a second feature of the motion data; an motion class identification step in which an motion classifier determines an estimated motion class membership probability indicating an estimated probability of motion data belonging to one or more motion classes based on the second feature; a person class identification step in which a person classifier determines an estimated person class membership probability indicating an estimated probability of motion data belonging to one or more person classes based on the second feature; and an motion class identification loss evaluation step in which, for the motion data, an motion class identification loss evaluation value that becomes smaller the higher the similarity between the estimated motion class membership probability and a given teacher motion class membership probability is, the learning method includes: a person class discrimination loss evaluation step of determining a person class discrimination loss evaluation value that becomes smaller as the similarity between the estimated person class membership probability and a teacher person class membership probability based on a person prediction result obtained by the person prediction device according to claim 3 for the action data is higher; and a learning step of learning each parameter of the action feature extractor, the action class classifier and the person class classifier for the action class discrimination loss evaluation value and the person class discrimination loss evaluation value so that the action feature extractor reduces the action class discrimination loss evaluation value and increases the person class discrimination loss evaluation value, and so that the action class classifier reduces the action class discrimination loss evaluation value and the person class classifier reduces the person class discrimination loss evaluation value.

[0019] One aspect of the present invention is a program for causing a computer to function as the above-mentioned person feature extractor learning device or the above-mentioned action recognition learning device. Effect of the Invention

[0020] According to the present invention, even when an ideal data set for realizing human action recognition cannot be obtained, it is possible to learn an action recognition model that generalizes to a variety of people. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a functional block diagram showing an example of a person feature extractor learning device 100 according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram illustrating an example of a person prediction device 200 according to the present embodiment. [Diagram 3] FIG. 2 is a functional block diagram showing an example of an action discrimination learning apparatus 300 according to the present embodiment. [Figure 4] FIG. 4 is a functional block diagram showing an example of an action identification device 400 according to the present embodiment. [Diagram 5] 1 is a flowchart showing an example of the operation of the person feature extractor learning device 100. [Figure 6] 13 is a flowchart showing an example of the operation of the action discrimination learning device 300. [Figure 7] FIG. 2 is a schematic diagram showing a hardware configuration common to each device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] <Summary> First, an outline of this embodiment will be described. The present embodiment aims to provide a technology that is capable of learning an action recognition model even when the following three conditions are satisfied for an available training data set. -Person labels are not assigned to the motion data For each person, there are classes of actions for which data is missing. The dataset includes action classes for which labeled data exists (hereinafter referred to as "known action classes") and unknown action classes for which no labeled data exists (hereinafter referred to as "unknown action classes").

[0023] In the following description, supervised motion data is motion data to which a correct motion class label has been assigned. Unsupervised motion data is motion data to which a correct motion class label has not been assigned. Training data is data including supervised motion data and unsupervised motion data. This embodiment operates appropriately when training data is given, even if one or more of the following problems are inherent. The first problem is that the training data is not assigned a person label. The second problem is that there is a motion class for which motion data is missing for each person. The third problem is that the training data includes motion data of an unknown motion class.

[0024] In this embodiment, the first problem is addressed by predicting and identifying person labels for the entire training data, while the second and third problems are addressed by using the predicted person labels to perform person-independent discriminative learning and by identifying and identifying those belonging to an unknown action class among the unsupervised action data.

[0025] <Example of the configuration of a person feature extractor learning device> FIG. 1 is a functional block diagram showing an example of a person feature extractor learning device 100 according to the present embodiment. The person feature extractor learning device 100 is configured using an information processing device such as a personal computer or a server device. The person feature extractor learning device 100 includes a control unit 90 and a learning result storage unit 120. The control unit 90 is configured using a processor such as a CPU (Central Processing Unit) and a memory. The control unit 90 functions as a preprocessing unit 101, a person feature extractor 102, a contrast loss evaluation unit 103, and a learning unit 104 by the processor executing a program. Note that all or part of the functions of the control unit 90 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded in a computer-readable storage medium. The computer-readable storage medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a semiconductor storage device (e.g., an SSD: Solid State Drive), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunication line.

[0026] The person feature extractor learning device 100 is communicatively connected to the data storage unit 110. The data storage unit 110 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device, or a device capable of reading data from a recording medium such as a CD-ROM.

[0027] The preprocessing unit 101 receives a set of motion data, performs preprocessing on each piece of motion data, and generates preprocessed motion data. The preprocessing unit 101 outputs the generated preprocessed motion data to the person feature extractor 102.

[0028] The person feature extractor 102 receives motion data or preprocessed motion data as input, and outputs a first feature amount. The person feature extractor 102 operates based on a function having a parameter capable of outputting such a first feature amount. The first feature amount is a numerical vector representing the feature of the person from which motion data is obtained, and is specifically represented by a vector having n1-dimensional elements determined by an integer value n1. The person feature extractor 102 reads the parameters stored in the learning result storage unit 120, and outputs the first feature amount of the input motion data. Any function can be used as the function applied to the person feature extractor 102, as long as it is differentiable with respect to the parameters. In this embodiment, a neural network is used.

[0029] The contrast loss evaluation unit 103 receives as input the set of first feature amounts output by the person feature extractor 102. The contrast loss evaluation unit 103 outputs an evaluation value (hereinafter referred to as a "contrast loss evaluation value") representing the similarity of the set of first feature amounts. The evaluation value is a loss value that becomes smaller as the similarity becomes higher when the set of first feature amounts are extracted from the same motion data, and becomes smaller as the similarity becomes lower when the set of first feature amounts are not extracted from the same motion data.

[0030] The learning unit 104 receives as an input the contrast loss evaluation value obtained by the contrast loss evaluation unit 103. The learning unit 104 learns the parameters of the person feature extractor 102 so as to reduce the input value. The learning unit 104 records the parameters obtained by learning in the learning result storage unit 120.

[0031] <Example of the configuration of a person prediction device> FIG. 2 is a functional block diagram showing an example of the person prediction device 200 according to the present embodiment. The person prediction device 200 is configured using an information processing device such as a personal computer or a server device. The person prediction device 200 includes a control unit 91, a storage unit 230, and a person prediction result storage unit 220. The control unit 91 is configured using a processor such as a CPU and a memory. The control unit 91 functions as a person feature extractor 201 and a person predictor 202 by the processor executing a program. All or part of the functions of the control unit 91 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded in a computer-readable storage medium. The computer-readable storage medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (e.g., SSD), or a storage device such as a hard disk or a reaction table storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0032] The storage unit 230 and the person prediction result storage unit 220 are configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 230 may be configured using a storage device such as a person feature extractor learning device (e.g., the learning result storage unit 120). The storage unit 230 stores parameters (e.g., data stored in the learning result storage unit 120) as the learning results obtained by learning by the learning unit 104 of the person feature extractor learning device 100.

[0033] When the person feature extractor 201 receives a motion data set to be processed (motion data set to be predicted), it reads parameters from the storage unit 230 and operates based on the parameters. The person feature extractor 201 outputs a set of first feature amounts for the motion data set to be processed. The person predictor 202 receives as input the set of first feature amounts obtained by the person feature extractor 201, and outputs a person prediction result which is a set of results predicting which person has generated the motion data for each motion data of the motion data set to be processed. The output person prediction result is recorded in the person prediction result storage unit 220.

[0034] <Configuration example of motion discrimination learning device> FIG. 3 is a functional block diagram showing an example of the action identification learning device 300 according to the present embodiment. The action identification learning device 300 is configured using an information processing device such as a personal computer or a server device. The action identification learning device 300 includes a control unit 92, an action identification result storage unit 350, and a learning result storage unit 340. The control unit 92 functions as an action feature extractor 301, an action class classifier 302, a person class classifier 303, an action class identification loss evaluation unit 304, a person class identification loss evaluation unit 305, and a learning unit 306 by a processor executing a program. All or part of the functions of the control unit 92 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded in a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (e.g., SSD), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0035] The motion identification learning device 300 operates by acquiring data from a supervised motion data storage unit 310, an unsupervised motion data storage unit 320, and a person prediction result storage unit 330. The supervised motion data storage unit 310 is configured using a device capable of reading motion data from a storage device such as a magnetic hard disk device or a semiconductor storage device, or a recording medium such as a CD-ROM. The supervised motion data storage unit 310 stores a supervised motion data set. The supervised motion data set is a set of motion data to which a desired motion class label is assigned. The unsupervised motion data storage unit 320 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device, or a recording medium such as a CD-ROM. The unsupervised motion data storage unit 320 stores an unsupervised motion data set. The unsupervised motion data set is a set of motion data to which a desired motion class label is not assigned.

[0036] The motion feature extractor 301 receives a supervised motion data set and an unsupervised motion data set as input, and extracts a second feature from each motion data. The motion feature extractor 301 outputs the extracted second feature to the motion class classifier 302 and the person class classifier 303. The second feature represents the feature of the required motion data as a vector having n2-dimensional elements, where n2 is an arbitrary integer. Note that the second feature is described as having a vector format for convenience, but the format is irrelevant to the gist of the present invention and may take any format. The motion feature extractor 301 reads parameters stored in the learning result storage unit 340, and outputs the second feature.

[0037] The action class identifier 302 receives as an input the second feature value output by the action feature extractor 301. The action class identifier 302 outputs an estimated action class belonging probability based on the input second feature value. The estimated action class belonging probability is the probability that the action data belongs to each known action class and unknown action class. The action class identifier 302 operates based on a function having parameters capable of outputting such an estimated action class belonging probability. The action class identifier 302 reads the parameters stored in the learning result storage unit 340 and outputs the estimated action class belonging probability. In addition, the action class identifier 302 identifies the associated action class from the estimated action class belonging probability and records the identification result in the action identification result storage unit 350.

[0038] The person classifier 303 receives as an input the second feature amount output by the action feature extractor 301. The person classifier 303 outputs an estimated person class membership probability based on the second feature amount. The estimated person class membership probability is a probability indicating which person's action the action data is made by. The person classifier 303 operates based on a function having parameters capable of outputting such an estimated person class membership probability. The person classifier 303 reads the parameters stored in the learning result storage unit 340 and outputs the estimated person class membership probability.

[0039] Any function can be used as long as it is differentiable with respect to the parameters for the action feature extractor 301, the action class classifier 302, and the person class classifier 303. In this embodiment, a neural network is used.

[0040] The action class identification loss evaluation unit 304 receives as input the estimated action class belonging probability output by the action class identifier 302 for the action data to be processed, the pseudo teacher action class belonging probability generated based on the action class identification result stored in the action identification result storage unit 350, and the teacher action class belonging probability representing the belonging probability to a desired action class. The pseudo teacher action class belonging probability is a tentative teacher action class belonging probability assigned to unsupervised action data to which a teacher-attributed action class is not assigned. The action class identification loss evaluation unit 304 calculates the value of an action class identification loss function (hereinafter referred to as "action class identification loss evaluation value"), which is a first loss evaluation function representing the difference between the estimated action class belonging probability and the pseudo teacher action class belonging probability or the teacher action class belonging probability.

[0041] The person class identification loss evaluation unit 305 receives as input the teacher person class membership probability based on the person prediction result stored in the person prediction result storage unit 330 and the estimated person class membership probability output by the person classifier 303 for the motion data to be processed. The person class identification loss evaluation unit 305 calculates a value of a person class identification loss function (hereinafter referred to as a "person class identification loss evaluation value") which is a second loss function representing the difference between the estimated person class membership probability and the teacher person class membership probability.

[0042] The learning unit 306 receives as input the action class identification loss evaluation value obtained by the action class identification loss evaluation unit 304 and the person class identification loss evaluation value obtained by the person class identification loss evaluation unit 305. The learning unit 306 uses the input values ​​to learn parameters of the action feature extractor 301, the action class classifier 302, and the person class classifier 303. The learning unit 306 records the parameters of the action feature extractor 301, the action class classifier 302, and the person class classifier 303 obtained by learning in the learning result storage unit 340. Learning for the action feature extractor 301 is performed so as to reduce the action class identification loss evaluation value and to increase the person class identification loss evaluation value. Learning for the action class classifier 302 is performed so as to reduce the action class identification loss evaluation value. Learning for the person class classifier 303 is performed so as to reduce the person class identification loss evaluation value.

[0043] <Configuration example of action recognition device> FIG. 4 is a functional block diagram showing an example of an action identification device 400 according to the present embodiment. The action identification device 400 is configured using an information processing device such as a personal computer or a server device. The action identification device 400 includes a control unit 93 and a storage unit 430. The control unit 93 is configured using a processor such as a CPU and a memory. The control unit 93 functions as an action feature extractor 401 and an action class identifier 402 by the processor executing a program. Note that all or part of the functions of the control unit 93 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded in a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (e.g., SSD), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0044] The storage unit 430 is configured using a storage device such as a magnetic hard disk device, a semiconductor storage device, etc. The storage unit 430 stores parameters (e.g., information stored in the learning result storage unit 340) obtained as a learning result by the learning performed by the learning unit 306 of the action discrimination learning device 300.

[0045] When the action feature extractor 401 receives the action data 410 to be processed (action data to be predicted), it reads out parameters from the storage unit 430 and operates based on the parameters. The action feature extractor 401 outputs a second feature amount for the action data 410 to be processed. The action class identifier 402 reads out parameters from the storage unit 430 and operates based on the parameters. The action class identifier 402 calculates an estimated action class membership probability for the action data 410 to be processed based on the second feature amount obtained by the action feature extractor 401. The output of the action class identifier 402 may be the membership probability itself for each action class for the action data 410 to be processed, or may be information indicating an estimation result of which action class the action data 410 belongs to.

[0046] <Example of operation of the person feature extractor learning device> 5 is a flowchart showing an example of the operation of the person feature extractor training device 100. Next, a description will be given of an example of the operation of the person feature extractor training device 100. The person feature extractor training device 100 receives a set of motion data from the data storage unit 110, and executes a learning process routine shown in FIG.

[0047] First, the control unit 90 of the person feature extractor learning device 100 reads two or more motion data sets (step S101). Next, the control unit 90 applies preprocessing to each of the motion data in the input motion data sets to generate preprocessed motion data (step S102).

[0048] The person feature extractor 102 and the contrast loss evaluation unit 103 execute processing on the input action data and the preprocessed action data generated in step S102, thereby obtaining a contrast loss evaluation value (step S103).

[0049] When the process of step S103 ends, the learning unit 104 updates the parameter values ​​of the person feature extractor 102 (values ​​to be recorded in the learning result storage unit 120) (step S104).

[0050] Then, the control unit 90 determines whether or not the end condition is satisfied (step S105). If the end condition is satisfied (step S105-YES), the control unit 90 ends the process. On the other hand, if the end condition is not satisfied (step S105-NO), the control unit 90 returns to step S101 and repeats the process.

[0051] The above-described iterative learning learns the parameters of the person feature extractor 102. The person feature extractor 102 uses the contrast loss evaluation value to perform learning so as to reduce the contrast loss evaluation value.

[0052] [Details of each process] Next, the processing of each processing unit of the person feature extractor training device 100 will be described in detail. The pre-processing of the motion data in step S102 will now be described in detail.

[0053] The objective of the person feature extractor learning device 100 is to learn the person feature extractor 102 from training data to which a person label is not given. Since a person label is not given, it is impossible to learn the person feature extractor 102 in a so-called general supervised learning framework. On the other hand, if the motion data is the same, at least the series of motions can be considered to be motions performed by the same person. Therefore, in the embodiment of the present invention, it is considered to learn the person features by performing some preprocessing on each motion data to duplicate the motion data, and learning to make the feature amounts extracted from the original motion data and the duplicated motion data closer to each other.

[0054] The problem here is that since motion data naturally contains not only information about people but also information about motion, simply duplicating the data does not necessarily allow learning of features that capture only information that identifies people. Therefore, in the embodiment of the present invention, this problem is solved by introducing a preprocessing method that loses information that identifies motions while preserving information that identifies people.

[0055] There are various pre-processing methods that can satisfy such requirements. For example, the average value in the time direction may be calculated for the motion data (or some of its frames). Assume that the motion data is a time series of the positions of each joint of a person during a motion, such as motion capture data. In this case, time series information is lost by taking the average value in the time direction. Therefore, the information of the motion itself (information that identifies the motion) is lost. However, the time average value of the joint positions still contains information that represents the physical characteristics of the person, such as the height and proportions. Therefore, the above requirement is satisfied. Note that the average value does not have a dimension in the time direction. For example, if the original motion data is a recording of a motion expressed in d dimensions for each frame for T frames, the average value becomes a d-dimensional vector. Therefore, the dimensions no longer match those of the original motion data, making direct comparison impossible. However, comparison becomes possible by adjusting the dimensions through a simple process such as duplicating the average value in the time direction for the number of frames (T frames) of the original motion.

[0056] Another example is a process of randomly rearranging the chronological order of the action frames of the action data. As with taking the average value, the chronological information of the action (information that identifies the action) is lost by rearranging the order. However, the physical characteristics of the subject of the action may be preserved. Alternatively, a portion of the action frames of the action data may be randomly selected and extracted. In this case, the same effect can be obtained. In addition to this, there are various other possible forms of preprocessing. Any form can be used as preprocessing as long as it meets the above requirements.

[0057] The person feature extractor training device 100 performs a training process using the motion data that has been preprocessed as described above, or the original motion data.

[0058] [Processing of comparative loss evaluation section] The person feature extractor 102 receives the motion data x and outputs a vector z, which is a first feature amount, and calculates parameters θ z This is realized by a function that has the following properties:

[0059] The number of data in the action data set read in step S101 is set to N. The preprocessing unit 101 and the person feature extractor 102 are applied to the action data set read in, and a set of vectors is output for each action data, thereby obtaining 2N vectors. In this set of vectors, a certain vector z j Vector z that shares the original motion data with i All other vectors are vectors z k Vector z k There are 2(N-1) vectors. In this contrastive learning, vectors extracted from the same action data are trained to have a high similarity, and vectors generated from different action data are trained to have a low similarity, so that action data of the same person are trained to be located close to each other in the vector z space. An example of a contrastive loss that realizes this type of learning is the following formula 1, which uses sim(u, v) to measure the similarity between vectors.

[0060]

number

[0061] Eb[a] is the expected value for the probability b of a. In this embodiment, Equation 1 is approximately replaced by the average for N pieces of motion data as shown in Equation 2 below. i and z i+1 is a first feature extracted from the same motion data.

[0062]

number

[0063] Equation 2 is expressed as θ z By reducing θ, a desirable person feature extractor 102 can be obtained. z There are various ways to calculate θ. In a simple way, the function that realizes the person feature extractor 102 has parameters θ z It is known that if the function is differentiable with respect to x, then it can be locally minimized. Therefore, in one example of this embodiment, a function that satisfies the following condition may be selected for the person feature extractor 102.

[0064] Conditions for the person feature extractor 102 A function that receives motion data x and outputs a feature vector z of that motion data. θ z Differentiable with respect to

[0065] [Learning process] The learning process of the learning unit 104 in step S104 will be described. The learning unit 104 calculates a contrast loss evaluation value L ctr The learning process is performed so that is small.

[0066] [Parameter storage process] After the parameter learning, the learning unit 104 learns the parameter θ z in the learning result storage unit 120. The learning unit 104 repeats the learning process from steps S101 to S104 described above until a termination condition is satisfied. Any condition may be used as the termination condition. For example, the termination condition may be that a predetermined number of repetitions are performed, or that the comparative loss evaluation value does not change by more than a certain value.

[0067] <Example of the person prediction device in action> An example of the operation of the person prediction device 200 will be described. First, the control unit 91 of the person prediction device 200 reads all of the motion data from the data storage unit 210. The person feature extractor 201 of the control unit 91 reads the learning result parameters recorded by the person feature extractor learning device 100 from the storage unit 230. The person feature extractor 201 processes the set of input motion data based on the learning result parameters, and obtains a set of first feature amounts extracted for each motion data.

[0068] The person predictor 202 obtains a person prediction result for each piece of action data based on the first feature amount. The person predictor 202 records the obtained person prediction result in the person prediction result storage unit 220.

[0069] [Details of person prediction process] The person feature extractor 201 outputs a first feature amount for each input action data. The obtained first feature amount is located in the feature amount space in such a way that action data features of the same person are located close to each other. The person predictor 202 executes a clustering algorithm on this set of features to obtain a person prediction result.

[0070] Any known clustering algorithm may be used as the clustering algorithm executed by the person predictor 202. For example, a clustering algorithm provided by open source software such as Scikit-learn may be used.

[0071] <Example of the motion discrimination learning device> Fig. 6 is a flowchart showing an operation example of the action discrimination learning device 300. Next, a description will be given of an operation example of the action discrimination learning device 300. The action discrimination learning device 300 reads data from the supervised action data storage unit 310, the unsupervised action data storage unit 320, and the person prediction result storage unit 330, and executes the learning process shown in Fig. 6.

[0072] The control unit 92 of the action identification learning device 300 reads one or more supervised action data sets, unsupervised action data sets, and person prediction result sets for each action data (step S201). Next, if the number of learning iterations is equal to or less than the scheduled number (step S202-YES), the control unit 92 executes the process of step S203. On the other hand, if the number of iterations is greater than the scheduled number (step S202-NO), the control unit 92 executes the process of step S204.

[0073] Here, the significance of the branching process in step S202 will be described. The purpose of the branching process is to have the action class identifier 302 learn to identify unknown action classes for which no teacher label exists. The action class identifier 302 learns to be able to identify (K+1) action classes, which are a combination of K known action classes and an unknown action class.

[0074] However, for known action classes, action data with action class labels exists in the supervised action data set, but for unknown action classes, there is no action data with a label indicating that the action classes are unknown. Therefore, if learning is performed using only the given action class labels, it is possible to identify K known action classes, but it is not possible to identify unknown action classes. Therefore, if the number of iterations is less than the expected number, the unsupervised action data set is classified as an unknown action class based on the output result of the action class identifier 302, and this is recorded in the classification history as the classification result of the (K+1)th class.

[0075] Thereafter, if the number of iterations is greater than the expected number, a pseudo-generated action class label is generated based on the action class classification results recorded in the classification history, and the action class classifier 302 is trained to be able to classify (K+1) classes. The action class classification results are recorded in the classification history again, and are used to generate action class labels during the next iterative learning.

[0076] By using a method for classifying the unknown action class based on the output result of the action class classifier 302 used when the number of iterations is equal to or less than the expected number, it is technically possible to classify the unknown action class. However, there is a problem that this method alone contains many errors. Therefore, by generating pseudo teacher action class labels including the unknown action class for the unsupervised action data, and then performing training of the action class classifier 302 and generation of the unknown action class labels alternately, it is possible to realize unknown action class classification with fewer errors.

[0077] In step S203, the action feature extractor 301, the action class classifier 302, and the person class classifier 303 perform processing on the supervised action data set, the unsupervised action data set, and the person prediction result set to obtain an action class classification loss evaluation value, a person class classification loss evaluation value, and an action class classification result.

[0078] In step S204, for the unsupervised action data set, an action class classification history is read from the action classification result storage unit 350. Then, in step S205, the action feature extractor 301, the action class classifier 302, and the person classifier 303 execute processing on the supervised action data set, the unsupervised action data set, the person prediction result set, and the action class classification history, thereby obtaining an action class classification loss evaluation value, a person class classification loss evaluation value, and an action class classification result.

[0079] After the process of step S203 or S205 is completed, the learning unit 306 updates the parameter values ​​(values ​​to be recorded in the learning result storage unit 340) of the action feature extractor 301, the action class classifier 302, and the person class classifier 303 based on the action class classification loss evaluation value and the person class classification loss evaluation value (step S206). The learning unit 306 stores the parameters of the action feature extractor 301, the action class classifier 302, and the person class classifier 303 in the learning result storage unit 340. Next, the action class classifier 302 records the action class classification result obtained in step S203 or S205 in the action classification result storage unit 350 (step S207).

[0080] Then, the control unit 92 determines whether the end condition is satisfied (step S208). If the end condition is satisfied (step S208-YES), the control unit 92 ends the process. If the end condition is not satisfied (step S208-NO), the control unit 92 returns to step S201 and repeats the process.

[0081] By the iterative learning described above, the parameters of the action feature extractor 301, the action class discriminator 302, and the person class discriminator 303 are learned. The action feature extractor 301 is learned to reduce the action class discrimination loss evaluation value and increase the person discrimination loss evaluation value. The action class discrimination loss function outputs a smaller value as the similarity between the estimated action class belonging probability of the action data output by the action class discriminator 302 and the given teacher action class belonging probability of the action data or the pseudo teacher action class belonging probability generated using the class discrimination history stored in the action discrimination result storage unit 350 increases. The person class discrimination loss function outputs a smaller value as the similarity between the estimated person action class of the action data output by the person class discriminator 303 and the person prediction result of the action data stored in the person prediction result storage unit 330 increases. In addition, the action class discriminator 302 learns to reduce the action class discrimination loss evaluation value, and the person class discriminator 303 learns to reduce the person class discrimination loss evaluation value.

[0082] [Details of each process] Next, the processing of each processing unit of action discrimination learning device 300 will be described in detail.

[0083] [If the number of iterations is less than the planned number] The processes of the action class classifier 302, the action class classification loss evaluator 304, and the person class classification loss evaluator 305 in step S202 when the number of iterations is equal to or less than the expected number of times will be described.

[0084] [Processing of motion discrimination loss evaluation part] The motion feature extractor 301 receives motion data x as input, outputs a vector f representing a second feature amount, and calculates parameters θ eThe action classifier 302 receives a vector f as an input and outputs an estimated membership probability y using a parameter θ m The estimated action class membership probability can be expressed as a probability function using a function F that realizes the action feature extractor 301, as shown in the following formula 3.

[0085]

number

[0086] Equation 3 is θ e , θ m , and the probability that y appears given x. The desired action feature extractor 301 and action class classifier 302 are such that when action data s is obtained from a supervised action data set, a teacher action class membership probability t for each action class appears. In other words, the action feature extractor 301 and action class classifier 302 are capable of determining the membership probability that allows identification of the correct action class. If the probability of appearance of the teacher action class membership probability t corresponding to the action data s is p(s, t), learning adjusts the parameter θ so that the value of Equation 4 becomes small. e , θ m It would be good if we could determine this.

[0087]

number

[0088] In this embodiment, the supervised motion data is given in the form of a set of pairs of motion data and teacher motion class membership probability. Therefore, the expected value is approximately replaced by the sum as shown in the following formula 5.

[0089]

number

[0090] Here, S and T are a set of one or more action data and the corresponding teacher action class membership probability. Equation 5 is an example of the action class classification loss function in this embodiment, and the value evaluated for any S and T is the classification loss function value.

[0091] Equation 5 is converted into θ e , θ m By reducing θ, it is possible to obtain a desirable motion feature extractor 301 and motion class classifier 302 that can output t for s. e , θ m There are various ways to calculate the function F that realizes the motion feature extractor 301 and the probability function that represents the motion class classifier 302. e , θ m It is known that local minimization is possible when the function is differentiable with respect to . Therefore, in one example of this embodiment, functions that satisfy the following conditions may be selected for the motion feature extractor 301 and the motion class identifier 302.

[0092] Conditions for the motion feature extractor 301 It is a function that outputs a feature vector f of the motion data when the motion data x is input. θ e Differentiable with respect to

[0093] Conditions for the behavior classifier 302 A function that takes a feature vector f as input and outputs an estimated action class membership probability y. θ y Differentiable with respect to

[0094] [Person class identification loss evaluation part processing] The person classifier 303 receives the feature vector f output by the motion feature extractor 301 and outputs the estimated person class membership probability using parameters θ c The estimated domain belonging probability can be expressed as a probability function as shown in the following formula 6 using a function F that realizes the motion feature extractor 301.

[0095]

number

[0096] Equation 6 is θ e , θ c , and the probability that c appears given x. A desirable person classifier 303 is one whose classification result for action data x given from a supervised action data set or an unsupervised action data set matches the teacher person class belonging probability given from the person prediction result set. For this reason, if the occurrence probability of the teacher person class belonging probability c corresponding to action data x is p(x, c), in the learning of the person classifier 303, the parameter θ is adjusted so that the following formula 7 becomes small. c It would be good if we could determine this.

[0097]

number

[0098] On the other hand, the motion feature extractor 301 adjusts the parameter θ so that Equation 7 becomes larger. e The system is trained to determine

[0099] The significance of the person class classifier 303 learning to reduce the value of formula 7, while the action feature extractor 301 learning to increase the value of the same formula 7, will be explained. The action feature extractor 301 of the present invention aims to make the classification by the action class classifier 302 less susceptible to the influence of differences between people by extracting invariant features about people. By learning to reduce formula 7, the person class classifier 303 becomes able to more accurately identify people. In contrast, by learning to increase it, the action feature extractor 301 outputs features that make it more difficult to identify people. By continuing this learning, the action feature extractor 301 becomes able to extract invariant features about people that are less susceptible to differences between people.

[0100] In this embodiment, Equation 7 can be approximately replaced in the form of a sum as shown in Equation 8 below, similar to Equation 4.

[0101]

number

[0102] Here, X and C are a set of one or more action data and the corresponding teacher person class membership probability. Equation 8 is a person class classification loss function in one example of this embodiment, and the value evaluated for any X and C is the classification loss function value.

[0103] Conditions for the motion feature extractor 301 It is a function that outputs a feature vector f of the motion data when the motion data x is input. θ e Differentiable with respect to

[0104] Conditions for the person classifier 303 - It is a function that takes a feature vector f as input and outputs an estimated person class membership probability y^. θ c Differentiable with respect to

[0105] [Classification process of the classifier] The probability that the action data x belongs to any of the known classes is the estimated action class membership probability y output by the action class identifier 302, and can be expressed by the above-mentioned formula 3. Regarding the estimated action class membership probability y output by the action class identifier 302, the information entropy H(y|x) indicating the ambiguity of the output estimated action class membership probability y is expressed by the following formula 9.

[0106]

number

[0107] Whether or not the unsupervised action data u in the unsupervised action data set is unknown class action data is identified by whether or not the value of the information entropy in Equation 9 is greater than a threshold value σ. Action data that is determined to be an unknown class based on the information entropy is identified as unknown class K+1, and other action data is identified as the class with the highest estimated action class belonging probability y of the action class identifier 302. In other words, the action class identification result y of the unsupervised action data u in the eth iteration is u、e is expressed as the following Equation 10.

[0108]

number

[0109] [If the number of iterations is greater than the planned number] Next, the process in step S204 will be described. In the process in step S204, the past classification results for each action data in the unsupervised action data set are read, and the probability of belonging to the pseudo teacher action class for the unsupervised action data is calculated from the past classification results. For the action data u in the unsupervised action data set, the action class classification result at the time of the iteration count t is calculated as y in step S207 described later. u、t In step S204, the past T classification results are read from the classification result storage unit 350, and the most frequent class in the past T is set as the pseudo teacher action class belonging probability. That is, the pseudo teacher action class probability y u is expressed as Equation 11 using mode(·) which represents the mode.

[0110]

number

[0111] Next, the evaluation process in step S205 will be described. The process performed by the action class identification loss evaluation unit 304 and the person class identification loss evaluation unit 305 is substantially the same as that performed in step S203 when the number of iterations is equal to or less than a certain number.

[0112] [Processing of the behavioral class discrimination loss evaluation part] The action class identification loss evaluation unit 304 calculates an action class identification loss function by taking the sum of the union of the set (S, T) of supervised action data and its supervised action class membership probabilities and the set (S', T') of unsupervised action data and the pseudo-supervised action class membership probabilities calculated in step 204. That is, the evaluation value of the action class identification loss evaluation unit 304 is expressed in the form of the following formula 12.

[0113]

number

[0114] [Person class identification loss evaluation part processing] The processing of the person class identification loss evaluation unit is the same as the processing of the person class identification loss evaluation unit 305 in step S203.

[0115] [Action class identification process of the action class classifier] In step S205, the action class identifier 302 can output the estimated action class membership probability of K+1 classes including the K known action classes and the unknown action class. The action class identification result of the action class identifier 302 is the class with the highest estimated action class membership probability among the K+1 classes. That is, the action class identification y of the unsupervised action data u at the iteration count e is u、e is expressed as the following Equation 13.

[0116]

number

[0117] [Learning process] The learning process of the learning unit 306 in step S206 will be described. s is small, and the person class identification loss evaluation value L cThe learning process is performed so that the value of the action class discrimination loss evaluation value L s The person classifier 303 learns so that the person class classification loss evaluation value becomes small. Specifically, this is done by sequentially optimizing the problems shown in Equation 14 and Equation 15.

[0118]

number

[0119]

number

[0120] Here, the class discrimination loss evaluation value L s and the domain discrimination loss evaluation value L c is the parameter θ e , θ m , θ c In order to select the functions of the action feature extractor 301, the action class classifier 302, and the person class classifier 303 so as to be differentiable with respect to , it is possible to learn them by the error gradient descent method.

[0121] [Parameter storage process] After the parameter learning, in the process of step S206, the parameter θ e , θ m , θ c is stored in the learning result storage unit 340.

[0122] [Learning result storage process] In the process of step S207, the action class identification result y u、e is stored in the action recognition result storage unit 350.

[0123] The learning process from steps S201 to S207 described above may be repeated until a termination condition is met. Any information may be used as the termination condition. For example, the termination condition may be that a predetermined number of repetitions are performed, that the value of the objective function does not change beyond a certain level, or that the accuracy of the evaluation data prepared separately from the training data does not change beyond a certain level.

[0124] <Operation of the motion recognition device> An example of the operation of the action identification device 400 will be described. First, the control unit 93 of the action identification device 400 reads the action data 410. Next, the action feature extractor 401 and the action class identifier 402 of the control unit 93 read the learning result parameters recorded by the action identification learning device 300 from the storage unit 430. The action feature extractor 401 and the action class identifier 402 that have read the parameters perform processing on the action data 410 to obtain the identification result 420.

[0125] Fig. 7 is a schematic diagram showing a hardware configuration common to each device. Each of the person feature extractor learning device 100, the person prediction device 200, the action identification learning device 300, and the action identification device 400 may be configured using an information processing device 900 as shown in Fig. 7. The information processing device 900 includes a processor 901, a memory 902, and an auxiliary storage device 903. The processor 901, the memory 902, and the auxiliary storage device 903 are communicatively connected via a bus 904.

[0126] In this embodiment configured as described above, even when a person label is not given to the training data, a problem is solved by predicting a person in the action data using the person prediction device 200. Even in cases where there is an inherent problem that there is an action class with missing action data for each person, or the training data contains action data of an unknown action class, the problem is solved by learning invariant features of a person using the predicted person label, and identifying and specifying unknown action data from action data to which no teacher label is given.

[0127] The data storage unit 110 may be provided in the person feature extractor learning device 100. The learning result storage unit 120 may be provided outside the person feature extractor learning device 100. Either or both of the data storage unit 210 and the person prediction result storage unit 220 may be provided in the person prediction device 200. The storage unit 230 may be provided outside the person prediction device 200. Either or more of the supervised action data storage unit 310, the unsupervised action data storage unit 320, and the person prediction result storage unit 330 may be provided in the action identification learning device 300. Either or both of the learning result storage unit 340 and the action identification result storage unit 350 may be provided outside the action identification learning device 300. The storage unit 430 may be provided outside the action identification device 400. When each storage unit is provided outside each device, the action data may be acquired by performing communication such as TCP / IP.

[0128] Each of the person feature extractor learning device 100, the person prediction device 200, the action identification learning device 300, and the action identification device 400 may be implemented using one information processing device, or may be implemented in a distributed manner across multiple information processing devices.

[0129] The person prediction device 200 may not include the person prediction result storage unit 220, but may be communicatively connected to a storage device that functions as the person prediction result storage unit 220. In this case, the person prediction device 200 and the person prediction result storage unit 220 are communicatively connected by wired communication or wireless communication.

[0130] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0131] 100...person feature extractor learning device, 101...preprocessing unit, 102...person feature extractor, 103...contrast loss evaluation unit, 104...learning unit, 110...data storage unit, 120...learning result storage unit, 200...person prediction device, 201...person feature extractor, 202...person predictor, 210...data storage unit, 220...person prediction result storage unit, 230...storage unit, 300...action identification learning device, 301...action feature extractor, 302...action class classifier, 303 ...person class classifier, 304...action class classification loss evaluation unit, 305...person class classification loss evaluation unit, 306...learning unit, 310...supervised action data storage unit, 320...unsupervised action data storage unit, 330...person prediction result storage unit, 340...learning result storage unit, 350...action classification result storage unit, 400...action classification device, 401...action feature extractor, 402...action class classifier, 410...action data, 420...classification result, 430...storage unit

Claims

1. a pre-processing unit that receives as input motion data that is data recording a time series of motions, and generates pre-processed motion data by performing a process on the motion data to eliminate information that identifies the motion; a person feature extractor that extracts a first feature from the motion data and the preprocessed motion data; a comparison loss evaluation unit that outputs a comparison loss evaluation value that, when a set of the first features are input, is smaller as the similarity of the set of first features increases when the set of first features are extracted from the same motion data, and is smaller as the similarity of the set of first features decreases when the set of first features are extracted from mutually different motion data; a learning unit that learns parameters of the person feature extractor so as to reduce the contrast loss evaluation value; A person feature extractor learning device comprising:

2. The person feature extractor learning device of claim 1 , wherein the pre-processing unit receives motion data as input and generates the pre-processed motion data by taking an average of a time series, or by taking a randomized order of the time series, or by taking a partial time series.

3. A person feature extractor which receives as input motion data which is data recording a time series of motions, generates preprocessed motion data by processing the motion data to remove information identifying the motion, extracts first features from the motion data and the preprocessed motion data, and outputs first features of the input motion data using features capturing only information identifying the person in the input motion data, using parameters which have been trained so that when a set of the first features are input, if the set of first features are extracted from the same motion data, the higher the similarity of the set of first features is, the smaller the contrast loss evaluation value becomes, and if the set of first features are extracted from different motion data, the lower the similarity of the set of first features is, the smaller the contrast loss evaluation value becomes. a person predictor for predicting a person of the action data based on the first feature amount; A person prediction device comprising:

4. a pre-processing step of receiving as input motion data, which is data recording a time series of motions, and generating pre-processed motion data by performing a process on the motion data to eliminate information identifying the motion; a person feature extraction step of extracting a first feature amount by a person feature extractor based on the motion data and the preprocessed motion data; an evaluation step of outputting a contrast loss evaluation value which, when a set of the first feature amounts is input, becomes smaller as the similarity of the set of first feature amounts increases if the set of first feature amounts are extracted from the same motion data, and which becomes smaller as the similarity of the set of first feature amounts decreases if the set of first feature amounts are extracted from mutually different motion data; a learning step of learning parameters of the person feature extractor so as to reduce the contrast loss evaluation value; A learning method that has

5. a motion feature extractor that receives motion data, which is data that records a time series of motions, as an input and extracts a second feature from the motion data; an action class classifier that calculates an estimated action class membership probability indicating an estimated membership probability to one or more action classes based on the second feature; a person class classifier that calculates an estimated person class membership probability indicating an estimated probability of belonging to one or more person classes based on the second feature amount; an action class identification loss evaluation unit that calculates an action class identification loss evaluation value that becomes smaller as the similarity between the estimated action class membership probability and a given teacher action class membership probability becomes higher; a person class identification loss evaluation unit that calculates a person class identification loss evaluation value that becomes smaller as the similarity between the estimated person class membership probability and a teacher person class membership probability based on a person prediction result obtained by the person prediction device according to claim 3 becomes higher; a learning unit that learns parameters of the action feature extractor, the action class classifier, and the person class classifier so that the action feature extractor decreases the action class classification loss evaluation value and increases the person class classification loss evaluation value, the action class classifier decreases the action class classification loss evaluation value, and the person class classifier decreases the person class classification loss evaluation value; A motion discrimination learning device comprising:

6. 6. The action identification learning device according to claim 5, wherein the action class classifier determines an unknown action class based on an action class membership probability obtained from the action class classifier when a number of iterative learnings in the learning section is equal to or less than a predetermined number of times.

7. the action class identifier further acquires a membership probability of an unknown action class when the number of iterative learning in the learning unit is greater than a predetermined number of times; 7. The action classification learning device according to claim 5, wherein the action class classification loss evaluation unit further obtains an action classification loss evaluation value that indicates a smaller value as a degree of similarity between the past action classification result and the unknown action class membership probability is higher, by using data to which no teacher label is assigned, a past action classification result, and the unknown action class membership probability.

8. a motion feature extractor that outputs a second feature amount using features capturing only information identifying a motion while excluding information identifying a person from input data, based on a parameter obtained by the motion discrimination learning device according to any one of claims 5 to 7; an action classifier that obtains a probability of the data belonging to a known action class and an unknown action class by using features that capture only information that identifies an action while excluding information that identifies a person from the data, based on the parameters obtained by the action identification learning device and the second feature amount; A motion identification device comprising:

9. a feature extraction step of receiving as input motion data, which is data recording a time series of motions, and outputting a second feature amount of the motion data by a motion feature extractor; an action class identification step of calculating an estimated action class membership probability indicating an estimated membership probability of the action data to one or more action classes by an action class classifier based on the second feature amount; a person class identification step of calculating an estimated person class membership probability indicating an estimated membership probability of the action data to one or more person classes based on the second feature amount by a person classifier; an action class identification loss evaluation step of calculating an action class identification loss evaluation value that becomes smaller as the similarity between the estimated action class membership probability and a given teacher action class membership probability for the action data increases; a person class identification loss evaluation step of determining a person class identification loss evaluation value that becomes smaller as the degree of similarity between the estimated person class membership probability and a teacher person class membership probability based on a person prediction result obtained by the person prediction device according to claim 3 increases for the action data; a learning step of learning each parameter of the action feature extractor, the action class classifier, and the person class classifier so that the action feature extractor decreases the action class classification loss evaluation value and increases the person class classification loss evaluation value, the action class classifier decreases the action class classification loss evaluation value, and the person class classifier decreases the person class classification loss evaluation value; A learning method that has

10. A program for causing a computer to function as the person feature extractor learning device according to claim 1 or 2, or the action identification learning device according to any one of claims 5 to 7.

11. An information system including a person feature extractor learning device and a person prediction device, The person feature extractor learning device, a pre-processing unit that receives as input motion data that is data recording a time series of motions, and generates pre-processed motion data by performing a process on the motion data to eliminate information that identifies the motion; a person feature extractor that extracts a first feature from the motion data and the preprocessed motion data; a comparison loss evaluation unit that outputs a comparison loss evaluation value that, when a set of the first features are input, is smaller as the similarity of the set of first features increases when the set of first features are extracted from the same motion data, and is smaller as the similarity of the set of first features decreases when the set of first features are extracted from mutually different motion data; a learning unit that learns parameters of the person feature extractor so as to reduce the contrast loss evaluation value; Equipped with The person prediction device includes: a person feature extractor that receives motion data as an input and outputs a first feature amount of the motion data based on a parameter obtained by the person feature extractor learning device according to claim 1 or 2; a person predictor for predicting a person of the action data based on the first feature amount; An information system comprising:

12. Further comprising a motion discrimination learning device, The motion discrimination learning device includes: a motion feature extractor that receives motion data, which is data that records a time series of motions, as an input and extracts a second feature from the motion data; an action class classifier that calculates an estimated action class membership probability indicating an estimated membership probability to one or more action classes based on the second feature; a person class classifier that calculates an estimated person class membership probability indicating an estimated probability of belonging to one or more person classes based on the second feature amount; an action class identification loss evaluation unit that calculates an action class identification loss evaluation value that becomes smaller as the similarity between the estimated action class membership probability and a given teacher action class membership probability becomes higher; a person class identification loss evaluation unit that calculates a person class identification loss evaluation value that becomes smaller as the similarity between the estimated person class membership probability and a teacher person class membership probability based on a person prediction result obtained by the person prediction device becomes higher; a learning unit that learns parameters of the action feature extractor, the action class classifier, and the person class classifier so that the action feature extractor decreases the action class classification loss evaluation value and increases the person class classification loss evaluation value, the action class classifier decreases the action class classification loss evaluation value, and the person class classifier decreases the person class classification loss evaluation value; The information system of claim 11 , comprising:

13. Further comprising a motion identification device, The motion identification device includes: a motion feature extractor that outputs a second feature amount of input data based on the parameters obtained by the motion discrimination learning device; an action classifier that acquires a probability of belonging to a known action class and an unknown action class for the data based on the parameters obtained by the action identification learning device and the second feature amount; The information system of claim 12 , comprising:

Images