Information processing device, information processing method, program

Abstract

It is difficult to accurately recognize movements from motion information related to localized body parts. [Solution] The information processing device of the present disclosure includes: a first motion calculation unit that calculates a first motion feature representing the characteristics of a person's movements based on whole-body motion data, which is motion data of the whole body of a person based on training data representing the movements of a person; a second motion calculation unit that calculates a second motion feature representing the characteristics of a person's movements based on local motion data, which is motion data of a local part of a person; and an update unit that updates parameters used by the second motion calculation unit to calculate the second motion feature based on training motion features, which are pre-specified motion characteristics of a person corresponding to training data, the first motion feature, and the second motion feature.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, for the purpose of automating employee work management and improving work efficiency, motion recognition technologies have been actively developed. For example, it is conceivable to visualize and analyze nursing tasks such as blood collection, eye drops, and meal delivery by nurses, in-store tasks such as cash register operation, product delivery, and cleaning by convenience store employees, and picking tasks by logistics warehouse workers.

[0003] Among motion recognition technologies, in particular, motion recognition technologies based on sensor data built into wearable devices such as acceleration, which have low system introduction costs, have attracted attention. Wearable devices represented by smartwatches are generally worn on a small number of body parts such as the wrist. For example, Non-Patent Document 1 describes predicting motion by inputting acceleration data from a wearable sensor worn on a body part.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since motion information from only the body part to which the device is attached makes it difficult to grasp the movements of the entire body, a problem arises in that it is difficult to recognize movements with high accuracy.

[0006] Therefore, one of the purposes of this disclosure is to solve the aforementioned problem of difficulty in accurately recognizing movements from motion information relating to localized body parts. [Means for solving the problem]

[0007] An information processing device, which is one form of this disclosure, A first motion calculation unit calculates a first motion characteristic that represents the characteristics of a person's movements based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, A second motion calculation unit calculates a second motion characteristic that represents the characteristics of a person's movements based on local motion data, which is motion data of a specific part of a person's body. An update unit updates the parameters used by the second motion calculation unit to calculate the second motion characteristics, based on the teacher motion characteristics, which are pre-identified motion characteristics of a person corresponding to the teacher data, the first motion characteristics, and the second motion characteristics. Equipped with, This is the structure it takes. Furthermore, the information processing method, which is one form of this disclosure, Information processing device, Based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, the first motion calculation unit calculates the first motion features that represent the characteristics of the person's movements. Based on the local motion data, which is motion data for specific parts of a person, the second motion calculation unit calculates a second motion characteristic that represents the characteristics of the person's movements. Based on the teacher motion features, which are pre-identified motion characteristics of a person corresponding to the aforementioned training data, the first motion features, and the second motion features, the parameters used by the second motion calculation unit to calculate the second motion features are updated. This is the structure it takes. Furthermore, one form of this disclosure is a program, In an information processing device, Based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, the first motion calculation unit calculates the first motion features that represent the characteristics of the person's movements. Based on the local motion data, which is motion data for specific parts of a person, the second motion calculation unit calculates a second motion characteristic that represents the characteristics of the person's movements. Based on the teacher motion features, which are pre-identified motion characteristics of a person corresponding to the aforementioned training data, the first motion features, and the second motion features, the parameters used by the second motion calculation unit to calculate the second motion features are updated. To execute the process This is the structure it takes. [Effects of the Invention]

[0008] This disclosure, configured as described above, enables highly accurate recognition of movements from movement information relating to a person's body parts. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example of the configuration of the information processing device related to this disclosure. [Figure 2] This flowchart shows an example of the processing operation of the information processing device related to this disclosure. [Figure 3] This is a block diagram showing an example of the configuration of the information processing device related to this disclosure. [Figure 4] This flowchart shows an example of the processing operation of the information processing device related to this disclosure. [Figure 5] This block diagram shows an example of the hardware configuration of the information processing device related to this disclosure. [Figure 6] This is a block diagram showing an example of the configuration of the information processing device related to this disclosure. [Modes for carrying out the invention]

[0010] <First Embodiment> The first embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any embodiment.

[0011] [Configuration] The information processing apparatus 10 of the present disclosure is used, for example, to generate a model for recognizing the actions of a person. In particular, the information processing apparatus 10 in the present embodiment is configured to generate a model for recognizing the whole-body motion from the motion data of a body part such as the acceleration data of a person's wrist. At this time, as will be described later, the information processing apparatus 10 trains the model using the whole-body motion data such as the joint coordinates of the human skeleton and the local motion data, and the model learns the dependency relationship between the whole-body motion and the local motion, so that high-precision motion recognition considering the whole-body motion can be performed according to the input of only the motion data of the body part.

[0012] Examples of the actions of the person to be recognized include, for example, nursing operations such as blood collection, eye drops, and meal delivery by nurses, in-store operations such as cash register operations, product picking, and cleaning by convenience store employees, and picking operations by logistics warehouse workers. However, the actions of the person to be recognized are not limited to the above-described actions and can be any action.

[0013] Fig. 1 shows the configuration of the information processing apparatus 10 in the present embodiment. The information processing apparatus 10 is composed of one or more information processing apparatuses including an arithmetic unit and a storage unit. And, as shown in Fig. 1, the information processing apparatus 10 includes a teacher modality action recognition unit 11, a target modality action recognition unit 12, a difference calculation unit 13, and a parameter update unit 14. The teacher modality action recognition unit 11 includes a modality conversion unit 15 and a whole-body motion feature extraction unit 16. The target modality action recognition unit 12 includes a local motion feature extraction unit 17 and a local motion prediction unit 18. The functions of the above-described units 11 to 18 can be realized by the arithmetic unit executing programs for realizing the respective functions stored in the storage unit. Hereinafter, each configuration will be described in detail. <0000​​The modality conversion unit 15 (conversion unit) included in the teacher modality motion recognition unit 11 (first motion calculation unit) acquires teacher modality full-body motion data D1, which is motion data of the whole body of a person, such as the joint coordinates of the person's skeleton. The teacher modality full-body motion data D1 is teacher data representing the motion of a person, the motion is specified in advance, and as will be described later, it is represented by a true motion vector D7. As the teacher modality full-body motion data D1, for example, there are RGB data of a video regarding a plurality of parts of a person's body, coordinate data of each joint of the skeleton, acceleration data, gyroscope data, and the like.

[0015] The modality conversion unit 15 generates target modality full-body motion data D2 (full-body motion data), such as acceleration at each joint, from the teacher modality full-body motion data D1. In particular, the target modality full-body motion data D2 is expressed in the same format as the target modality such as three-axis acceleration and does not hold motion information peculiar to the teacher modality such as joint coordinates. For example, the modality conversion unit 15 calculates to convert joint coordinate data of (number of joints × three-axis coordinates × number of time steps) dimensions into acceleration data of (number of joints × three-axis acceleration × number of time steps) dimensions, and removes information on the relative positional relationship of each joint peculiar to the joint coordinate data. By this process, it is possible to prevent the teacher modality motion recognition unit 11 from extracting full-body motion knowledge peculiar to the teacher modality from the teacher modality full-body motion data D1, and to extract full-body motion knowledge highly dependent on the target modality local motion data D2. Then, the modality conversion unit 15 outputs the target modality full-body motion data D2.

[0016] As an example, the modality conversion unit 15 inputs the teacher modality full-body motion data D1 using a generator of a pre-learned adversarial generation network to generate the target modality full-body motion data D2. Here, it is assumed that the adversarial generation network is pre-trained using whole-body joint coordinate data and whole-body acceleration data corresponding to each other.

[0017] Specifically, a generative adversarial network consists of a generator that takes whole-body joint coordinate data as input and generates whole-body acceleration data, and a discriminator that distinguishes the generated whole-body acceleration data from true whole-body acceleration data. The generator is trained to produce spurious acceleration data that the discriminator will misidentify as true acceleration data. In particular, the generated acceleration data is associated with label 1, input to the discriminator, and the difference is calculated using binary cross-entropy against the predicted output label to update the generator's parameters. On the other hand, the discriminator is trained to accurately distinguish between the acceleration data output by the generator and true acceleration data. In particular, the generated acceleration data corresponding to label 0 and the true acceleration data corresponding to label 1 are input to the discriminator, and the difference is calculated using binary cross-entropy against the predicted output label to update the discriminator's parameters.

[0018] In this way, by repeatedly training the generator and the discriminator alternately, the discriminator becomes able to distinguish subtle differences between the generated data and true acceleration data, and the generator becomes able to output generated data that is so similar to true acceleration data that it causes misclassification by a more advanced discriminator. For example, as a result of training, the generator can take skeletal joint coordinate data as input and generate acceleration data that takes into account the deviation of the accelerometer's measurement axis due to joint twisting. Note that the data used to train the generative adversarial network does not have to be the same as the data used to train the teacher modality motion recognition unit 11. For example, it does not have to be data acquired in the environment in which this information processing device is installed, but can be any open data.

[0019] The modality conversion unit 15 may also generate target modality whole-body motion data D2 from teacher modality whole-body motion data D1 by another method. For example, if the modality conversion unit 15 is given skeletal joint coordinate data as the teacher modality and acceleration data as the target modality, it will generate acceleration data by taking the second derivative of the displacement of the joint coordinates per unit time. Alternatively, if the modality conversion unit 15 is given RGB data of a video as the teacher modality and acceleration data as the target modality, it will extract the skeletal joint coordinates from each frame image of the video and generate acceleration data by taking the second derivative of the displacement of the joint coordinates per unit time. When the teacher modality and target modality are the same, the modality conversion unit 15 will treat the teacher modality whole-body motion data D1 as is and use it as target modality whole-body motion data D2.

[0020] The whole-body motion feature extraction unit 16, which is part of the teacher modality motion recognition unit 11 (first motion calculation unit), is a pre-trained neural network that acquires target modality whole-body motion data D2. The whole-body motion feature extraction unit 16 calculates whole-body motion feature data D3 (first motion feature, first feature quantity) from the target modality whole-body motion data D2, which represents the motion characteristics of the person's entire body. At this time, the whole-body motion feature extraction unit 16 calculates the whole-body motion feature data D3 so that it can be represented by a whole-body feature quantity, such as a 128-dimensional vector. The whole-body motion feature extraction unit 16 then outputs the calculated whole-body motion feature data D3.

[0021] In this way, the teacher modality motion recognition unit 11 calculates whole-body motion feature data D3, which represents the characteristic quantities of whole-body motion, as a characteristic of a person's motion, from the target modality whole-body motion data D2, which is obtained by converting the teacher modality whole-body motion data D1, which is motion data for the whole body of a person.

[0022] The local motion feature extraction unit 17 of the target modality motion recognition unit 12 (second motion calculation unit) is an untrained neural network that acquires target modality local motion data D4 (local motion data), which is motion data of a specific part of a person, such as acceleration data and gyroscope data of the person's wrist. At this time, the target modality local motion data D4 is measured at the same time and from the same person as the training modality whole-body motion data D1 described above. For example, the target modality local motion data D4 may be data acquired from a wearable device such as a smartwatch, or it may be data extracted from the target modality whole-body motion data D2 described above.

[0023] The local motion feature extraction unit 17 calculates local motion feature data D5 (second motion feature, second feature quantity) representing the characteristics of a person's motion from the target modality local motion data D4. At this time, the local motion feature extraction unit 17 calculates the feature quantity of the person's local motion data, but calculates the local motion feature data D5 in the same data format and with the same number of dimensions as the person's whole-body motion feature data D3 described above, for example, a 128-dimensional vector. Then, the local motion feature extraction unit 17 outputs the calculated local motion feature data D5.

[0024] The local motion prediction unit 18, which is part of the target modality motion recognition unit 12 (second motion calculation unit), is an untrained neural network that acquires local motion feature data D5. The local motion prediction unit 18 calculates a local predicted motion vector D6 (second motion feature, second predicted motion) that predicts the person's motion from the local motion feature data D5. The local predicted motion vector D6 is represented as a vector with the same number of dimensions as the number of motions of the recognition target, and the height of the value in each dimension represents the predicted probability of each corresponding motion. For example, the local predicted motion vector D6 is represented as [0.2,0.4,···,0.1]. The local motion prediction unit 18 then outputs the calculated local predicted motion vector D6.

[0025] In this way, the target modality motion recognition unit 12 calculates local motion feature data D5, which represents the feature quantities of the local motion, and local predicted motion vector D6, which represents the predicted motion of the person, from the target modality local motion data D4, which is motion data of a local part of the person. The target modality motion recognition unit 12 is a model composed of a neural network that predicts the motion of a person, and as will be explained below, the parameters of the neural network are trained and updated by machine learning.

[0026] The difference calculation unit 13 obtains the whole-body motion feature data D3, local motion feature data D5, and local predicted motion vector D6, which are calculated by the teacher modality motion recognition unit 11 and the target modality motion recognition unit 12 as described above. The difference calculation unit 13 also obtains the true motion vector D7 (teacher motion), which represents the motion of a pre-identified person corresponding to the teacher data, which is the teacher modality whole-body motion data D1. The true motion vector D7 is represented as a vector with the same number of dimensions as the number of motions set. For example, if three motions such as blood sampling, eye drops, and serving meals are set in nursing work, the true motion vector D7 is given as a one-hot vector such as [1,0,0], [0,1,0], and [0,0,1], where the elements of the corresponding motions are represented by "1". In this case, the number of dimensions of the true motion vector D7 is assumed to be the same as the number of dimensions of the local predicted motion vector D6 described above. In other words, the local motion prediction unit 18 described above calculates the local predicted motion vector D6 so that it is represented as a vector with the same data format and the same number of dimensions as the true motion vector D7.

[0027] The difference calculation unit 13 calculates the prediction error between the acquired whole-body motion feature data D3, local motion feature data D5, local predicted motion vector D6, and true motion vector D7, and outputs the calculated prediction error. Specifically, the difference calculation unit 13 first calculates the difference between the whole-body motion feature data D3 and the local motion feature data D5, then calculates the difference between the local predicted motion vector D6 and the true motion vector D7, and calculates the sum of these differences as the prediction error. In this case, for example, the difference between the whole-body motion feature data D3 and the local motion feature data D5 may be calculated using Euclidean distance, Manhattan distance, Mahalanobis distance, Chebyshev distance, etc. Also, the difference between the local predicted motion vector D6 and the true motion vector D7 may be calculated using cross-entropy, etc.

[0028] The parameter update unit 14 (update unit) acquires the prediction error and calculates update parameters from the prediction error. The update parameters are parameters set in the neural network that constitutes the local motion feature extraction unit 17 and the local motion prediction unit 18 provided in the target modality motion recognition unit 12 described above, and are used to calculate the local motion feature data D5 and the local predicted motion vector D6. Specifically, the parameter update unit 14 performs machine learning to calculate update parameters so that the prediction error is small. Then, the parameter update unit 14 outputs the calculated update parameters and sets them in the neural network that constitutes the local motion feature extraction unit 17 and the local motion prediction unit 18 of the target modality motion recognition unit 12.

[0029] In this way, by using machine learning to minimize the difference between whole-body motion feature data D3 and local motion feature data D5 included in the prediction error, it becomes possible to extract motion feature data D5 that appears to reflect whole-body motion from the target modality local motion data D4. Furthermore, by using machine learning to minimize the difference between the local predicted motion vector D6 and the true motion vector D7 included in the prediction error, the accuracy of motion prediction can be improved.

[0030] The target modality motion recognition unit 12, whose parameters have been updated by the machine learning described above, will then have the following functions during inference.

[0031] The local motion feature extraction unit 17 is a neural network trained by the machine learning described above, and acquires local motion data D4 of a target modality, such as wrist acceleration data, which is local motion data of a person. The local motion feature extraction unit 17 calculates and outputs local motion feature data D5 from the target modality local motion data D4. At this time, because the local motion feature extraction unit 17 was trained using the training modality whole-body motion data D1 as described above, it is able to extract motion features that reflect whole-body motion from the target modality local motion data D4.

[0032] The local motion prediction unit 18 is a pre-trained neural network trained by the machine learning described above, and acquires local motion feature data D5. The local motion prediction unit 18 calculates and outputs a local prediction motion vector D6 from the local motion feature data D5. The output local prediction motion vector D6 is represented as a vector with the same number of dimensions as the number of motions to be recognized, and the height of the value in each dimension represents the predicted probability of each corresponding motion. The motion with the highest probability value is then taken as the prediction result.

[0033] [Operation] Next, the operation of the information processing device 10 described above will be explained.

[0034] First, the teacher modality motion recognition unit 11 acquires teacher modality whole-body motion data D1 (step S1 in Figure 2). Next, the modality conversion unit 15 generates target modality whole-body motion data D2 from the teacher modality whole-body motion data D1 (step S2 in Figure 2). Subsequently, the whole-body motion feature extraction unit 16 extracts whole-body motion feature data D3 from the target modality whole-body motion data D2 (step S3 in Figure 2).

[0035] Next, the target modality motion recognition unit 12 acquires the target modality local motion data D4 (step S4 in Figure 2). Subsequently, the local motion feature extraction unit 17 extracts local motion feature data D5 from the target modality local motion data D4 (step S5 in Figure 2). Then, the local motion prediction unit 106 calculates the local predicted motion vector D6 from the local motion feature data D5 (step S6 in Figure 2).

[0036] Next, the difference calculation unit 13 obtains the true motion vector (step S7 in Figure 2). Then, the difference calculation unit 13 calculates the sum of the difference between the whole-body motion feature data D3 and the local motion feature data D4, and the difference between the local predicted motion vector D6 and the true motion vector D7 (step S8 in Figure 2). Next, the parameter update unit 14 uses machine learning to reduce the sum of the differences and calculates the update parameters for the target modality motion recognition unit 12 (step S9 in Figure 2). Finally, the parameter update unit 14 applies the calculated update parameters to the target modality motion recognition unit 12 (step S10 in Figure 2).

[0037] As a result, the information processing device 10 in this embodiment can extract motion characteristic data D5 that appears to reflect whole-body motion from target modality local motion data D4, thereby improving the accuracy of motion prediction from local motion data.

[0038] <Second Embodiment> Next, a second embodiment of this disclosure will be described with reference to the drawings. The drawings may be relevant to either embodiment.

[0039] [composition] The information processing device 10 in this embodiment has the same configuration as that described in Embodiment 1 above, but differs in the following respects. Below, we will mainly describe the configuration that differs from Embodiment 1, and omit the description of the configuration that is the same.

[0040] As shown in Figure 3, the information processing device 10 comprises a teacher modality motion recognition unit 11, a target modality motion recognition unit 12, a difference calculation unit 13, and a parameter update unit 14. The teacher modality motion recognition unit 11 also comprises a modality conversion unit 15, a whole-body motion feature extraction unit 16, and a whole-body motion prediction unit 19. The target modality motion recognition unit 12 also comprises a local motion feature extraction unit 17 and a local motion prediction unit 18. Each of the functions of the above-described units 11 to 19 can be realized by the arithmetic unit executing a program for realizing each function stored in the memory. The following describes each configuration in detail.

[0041] The modality conversion unit 15 (conversion unit) of the teacher modality motion recognition unit 11 (first motion calculation unit) acquires teacher modality whole-body motion data D1, which is motion data for the entire body of a person, such as the coordinates of each joint of the person's skeleton. Then, the modality conversion unit 15 generates target modality whole-body motion data D2, such as the acceleration at each joint, from the teacher modality whole-body motion data D1. The modality conversion unit 15 outputs the target modality whole-body motion data D2.

[0042] The whole-body motion feature extraction unit 16, which is part of the teacher modality motion recognition unit 11 (first motion calculation unit), is a pre-trained neural network that acquires target modality whole-body motion data D2. The whole-body motion feature extraction unit 16 then calculates whole-body motion feature data D2 from the target modality whole-body motion data D2. The whole-body motion feature data is represented, for example, as a 128-dimensional vector. The whole-body motion feature extraction unit 16 outputs whole-body motion feature data D3.

[0043] The whole-body motion prediction unit 19 (first motion calculation unit) of the teacher modality motion recognition unit 11 (first motion calculation unit) is a pre-trained neural network that acquires whole-body motion feature data D3. The whole-body motion prediction unit 17 calculates a whole-body predicted motion vector D8 (first motion feature, first predicted motion) that predicts the motion of a person from the whole-body motion feature data D3. The whole-body predicted motion vector D8 is represented as a vector with the same number of dimensions as the number of motions of the recognition target, and the height of the value in each dimension represents the predicted probability of each corresponding motion. For example, the whole-body predicted motion vector D8 is represented as [0.2,0.4,···,0.1]. In this case, the whole-body predicted motion vector D8 is represented as a vector with the same number of dimensions as the true motion vector D7. In other words, the whole-body motion prediction unit 19 calculates the whole-body predicted motion vector D8 so that it is represented as a vector with the same number of dimensions and the same data format as the true motion vector D7. The whole-body predicted motion vector D8 is also represented as a vector with the same number of dimensions and the same data format as the local predicted motion vector D6, which will be described later. The whole-body motion prediction unit 19 then outputs the calculated whole-body predicted motion vector D8.

[0044] In this way, the teacher modality motion recognition unit 11 calculates a whole-body predicted motion vector D8 as a characteristic of a person's motion from the teacher modality whole-body motion data D1, which is motion data for the whole body of a person.

[0045] The local motion feature extraction unit 17, which is part of the target modality motion recognition unit 12 (second motion calculation unit), is an untrained neural network that acquires target modality local motion data D4 (local motion data), which is motion data of a specific part of a person, such as acceleration data and gyroscope data of the person's wrist. At this time, the local motion data D4 is measured at the same time and from the same person as the training modality whole-body motion data D1 described above. The local motion feature extraction unit 17 calculates local motion feature data D5, which is the feature quantity of the motion data of a specific part of the person, from the target modality local motion data D4. The local motion feature data D5 is represented as a vector with the same number of dimensions as the whole-body motion feature data. The local motion feature extraction unit 17 then outputs the local motion feature data D5.

[0046] The local motion prediction unit 18, which is part of the target modality motion recognition unit 12 (second motion calculation unit), is an untrained neural network that acquires local motion feature data D5. The local motion prediction unit 18 calculates a local predicted motion vector D6 (second motion feature, second predicted motion) that predicts the motion of a person from the local motion feature data D5. The local predicted motion vector D6 is represented as a vector with the same number of dimensions as the number of motions of the recognition target, and the height of the value in each dimension represents the predicted probability of each corresponding motion. The local predicted motion vector D6 is represented as a vector with the same number of dimensions and data format as the true motion vector D7 and the whole-body predicted motion vector D8. The whole-body motion prediction unit 19 then outputs the calculated whole-body predicted motion vector D8.

[0047] In this way, the target modality motion recognition unit 12 calculates local motion feature data D5, which represents the feature quantities of the local motion of the person, and local predicted motion vector D6, which represents the predicted motion of the person, from the target modality local motion data D4, which is motion data of a local part of the person. The target modality motion recognition unit 12 is a model composed of a neural network that predicts the motion of a person, and as will be explained below, the parameters of the neural network are trained by machine learning.

[0048] The difference calculation unit 13 obtains the whole-body predicted motion vector D8 and the local predicted motion vector D6, which were calculated by the teacher modality motion recognition unit 11 and the target modality motion recognition unit 12 as described above. The difference calculation unit 13 also obtains the true motion vector D7 (teacher motion), which represents the motion of a pre-identified person corresponding to the teacher modality whole-body motion data D1, which is the teacher data.

[0049] The difference calculation unit 13 then calculates the prediction error between the acquired whole-body predicted motion vector D8, the local predicted motion vector D6, and the true motion vector D7, and outputs the calculated prediction error. Specifically, the difference calculation unit 13 first calculates the difference between the whole-body predicted motion vector D8 and the local predicted motion vector D6, then calculates the difference between the local predicted motion vector D6 and the true motion vector D7, and calculates the sum of these differences as the prediction error. In this case, for example, the difference between the whole-body predicted motion vector D8 and the local predicted motion vector D6 may be calculated using Euclidean distance, Manhattan distance, Mahalanobis distance, Chebyshev distance, cross-entropy, temperature-dependent cross-entropy, KL divergence, etc. Also, the difference between the local predicted motion vector D6 and the true motion vector D7 may be calculated using cross-entropy, etc.

[0050] The parameter update unit 14 (update unit) acquires the prediction error and calculates update parameters from the prediction error. The update parameters are parameters set in the neural network that constitutes the local motion feature extraction unit 17 and the local motion prediction unit 18 provided in the target modality motion recognition unit 12 described above, and are used to calculate the local motion feature data D5 and the local predicted motion vector D6. Specifically, the parameter update unit 14 performs machine learning to calculate update parameters so that the prediction error is small. Then, the parameter update unit 14 outputs the calculated update parameters and sets them in the neural network that constitutes the local motion feature extraction unit 17 and the local motion prediction unit 18 of the target modality motion recognition unit 12.

[0051] In this way, by using machine learning to minimize the difference between the whole-body predicted motion vector D8 and the local predicted motion vector D6 included in the prediction error, it becomes possible to predict motions that seem to reflect whole-body motion from the target modality local motion data D4. Furthermore, by using machine learning to minimize the difference between the local predicted motion vector D6 and the true motion vector D7 included in the prediction error, the accuracy of motion prediction can be improved.

[0052] Furthermore, the target modality motion recognition unit 12, whose update parameters have been updated by the machine learning described above, will have the same functions as in Embodiment 1 described above during inference. Specifically, when the target modality motion recognition unit 12 receives target modality local motion data D4, which is local motion data of a person, it calculates local motion feature data D5 from the target modality local motion data D4, and then calculates and outputs a local predicted motion vector D6 from the local motion feature data D5. The output local predicted motion vector D6 is represented as a vector with the same number of dimensions as the number of motions to be recognized, and the height of the value in each dimension represents the predicted probability of each corresponding motion. The motion with the highest probability value is then used as the prediction result.

[0053] [Operation] Next, the operation of the information processing device 10 described above will be explained.

[0054] First, the teacher modality motion recognition unit 11 acquires teacher modality whole-body motion data D1 (step S21 in Figure 4). Next, the modality conversion unit 15 generates target modality whole-body motion data D2 from the teacher modality whole-body motion data D1 (step S22 in Figure 4). Then, the whole-body motion feature extraction unit 16 extracts whole-body motion feature data D3 from the target modality whole-body motion data D2 (step S23 in Figure 4). Finally, the whole-body motion prediction unit 19 calculates a whole-body predicted motion vector D8 from the whole-body motion feature data D3 (step S24 in Figure 4).

[0055] Next, the target modality motion recognition unit 12 acquires the target modality local motion data D4 (step S25 in Figure 4). Subsequently, the local motion feature extraction unit 17 extracts local motion feature data D5 from the target modality local motion data D4 (step S26 in Figure 4). Then, the local motion prediction unit 18 calculates the local predicted motion vector D6 from the local motion feature data D5 (step S27 in Figure 4).

[0056] Next, the difference calculation unit 13 obtains the true motion vector (step S28 in Figure 4). Then, the difference calculation unit 13 calculates the sum of the difference between the whole-body predicted motion vector D8 and the local predicted motion vector D6, and the difference between the local predicted motion vector D6 and the true motion vector D7 (step S29 in Figure 4). Next, the parameter update unit 14 uses machine learning to reduce the sum of the differences and calculates the update parameters for the target modality motion recognition unit 12 (step S30 in Figure 4). Finally, the parameter update unit 14 applies the calculated update parameters to the target modality motion recognition unit 12 (step S31 in Figure 4).

[0057] As described above, the information processing device 10 in this embodiment can perform motion predictions that appear to reflect whole-body motion from the target modality local motion data D4, thereby improving the accuracy of motion predictions from local motion data.

[0058] <Third Embodiment> Next, a third embodiment of the present disclosure will be described with reference to the drawings. This embodiment shows an outline of the information processing device, etc., described in the embodiments described above. Note that the drawings may be relevant to any of the embodiments.

[0059] First, the hardware configuration of the information processing device 100 in this disclosure will be described. The information processing device 100 is composed of a general information processing device, and as an example, it is equipped with the following hardware configuration as shown in Figure 5. ·CPU(Central Processing Unit)101(Arithmetic unit) ROM (Read Only Memory) 102 (Storage Device) • RAM (Random Access Memory) 103 (Storage Device) • Program group 104 loaded into RAM 103 • Storage device 105 for storing the program group 104 • Drive device 106 for reading and writing to external storage medium 110 of the information processing device. • Communication interface 107 connecting to a communication network 111 outside the information processing device. • Input / output interface 108 for data input and output. • Bus 109 connecting each component

[0060] Figure 5 shows an example of the hardware configuration of the information processing device 100, and the hardware configuration of the information processing device is not limited to the case described above. For example, the information processing device may consist of only a part of the configuration described above, such as not having a drive device 106. In addition, the information processing device may use a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof instead of the CPU described above.

[0061] The information processing device 100 can be equipped with the first operation calculation unit 121, the second operation calculation unit 122, and the update unit 123 shown in Figure 6 by having the CPU 101 acquire the program group 104 and execute it. The program group 104 is, for example, stored in advance in a storage device 105 or ROM 102, and the CPU 101 loads it into RAM 103 and executes it as needed. The program group 104 may also be supplied to the CPU 101 via a communication network 111, or it may be stored in advance in a storage medium 110, and the drive device 106 reads the program and supplies it to the CPU 101. However, the first operation calculation unit 121, the second operation calculation unit 122, and the update unit 123 described above may be constructed with dedicated electronic circuits to realize such means.

[0062] The first motion calculation unit 121 calculates a first motion feature representing the characteristics of a person's movements based on whole-body motion data, which is motion data for the entire body of a person based on training data representing the person's movements. The second motion calculation unit 122 calculates a second motion feature representing the characteristics of a person's movements based on local motion data, which is motion data for local parts of a person. The update unit 123 updates the parameters used by the second motion calculation unit to calculate the second motion feature, based on the training motion feature, which is a pre-identified motion characteristic of a person corresponding to the training data, the first motion feature, and the second motion feature.

[0063] As described above, this disclosure enables motion prediction that reflects whole-body motion from motion data of localized parts of a person, thereby improving the accuracy of motion prediction from localized motion data.

[0064] Furthermore, at least one of the functions of the first operation calculation unit 121, the second operation calculation unit 122, and the update unit 123 described above may be executed on an information processing device installed and connected to any location on the network, that is, it may be executed using so-called cloud computing.

[0065] Furthermore, the aforementioned programs can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0066] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each of the embodiments described above can be combined with other embodiments as appropriate.

[0067] <Note> Some or all of the above embodiments may also be described as follows. The general configuration of the information processing apparatus, information processing method, and program in this disclosure is described below. However, this disclosure is not limited to the configurations described below. Furthermore, some or all of the configurations and functions described in Appendices 2 to 8, which are dependent on Appendice 1 below, may also be dependent on Appendices 9 and 10 in the same way as Appendices 2 to 8. Moreover, not limited to Appendices 1, 9, and 10, some or all of the configurations and functions described as appendices may also be dependent on similar hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above. (Note 1) A first motion calculation unit calculates a first motion characteristic that represents the characteristics of a person's movements based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, A second motion calculation unit calculates a second motion characteristic that represents the characteristics of a person's movements based on local motion data, which is motion data of a specific part of a person's body. An update unit updates the parameters used by the second motion calculation unit to calculate the second motion characteristics, based on the teacher motion characteristics, which are pre-identified motion characteristics of a person corresponding to the teacher data, the first motion characteristics, and the second motion characteristics. Equipped with an information processing device. (Note 2) The information processing device described in Appendix 1, The system includes a difference calculation unit that calculates the difference between the first motion feature and the second motion feature, and the difference between the teacher motion feature and the second motion feature. The update unit updates the parameters based on each of the calculated differences. Information processing device. (Note 3) The information processing device described in Appendix 2, The update unit updates the parameters so that the sum of the calculated differences becomes smaller. Information processing device. (Note 4) The information processing device described in Appendix 2, The first motion calculation unit calculates a first feature quantity that represents the characteristic quantity of motion in the whole body of a person based on the whole body motion data, as the first motion feature. The second motion calculation unit calculates, as the second motion features, a second feature quantity representing the motion characteristics of a local part of the person, and a second predicted motion representing the motion predicted by the person, based on the local motion data. The difference calculation unit calculates the difference between the first feature and the second feature, and the difference between the teacher action, which represents the identified action of a person that is a teacher action feature, and the second predicted action. Information processing device. (Note 5) The information processing device described in Appendix 4, The first motion calculation unit and the second motion calculation unit calculate the first feature quantity and the second feature quantity in the same data format. The second motion calculation unit calculates the second predicted motion in the same data format as the teacher motion. Information processing device. (Note 6) The information processing device described in Appendix 2, The first motion calculation unit calculates a first predicted motion, which represents the motion predicted by the person based on the whole-body motion data, as the first motion feature. The second motion calculation unit calculates a second predicted motion, which represents the motion predicted by the person based on the local motion data, as the second motion feature. The difference calculation unit calculates the difference between the first predicted motion and the second predicted motion, and the difference between the teacher motion, which represents the identified motion of the person that is a teacher motion feature, and the second predicted motion. Information processing device. (Note 7) The information processing device described in Appendix 6, The first motion calculation unit and the second motion calculation unit calculate the first predicted motion and the second predicted motion in the same data format as the teacher motion. Information processing device. (Note 8) The information processing device described in Appendix 1, The first motion calculation unit includes a conversion unit that converts the training data, which consists of coordinate data of each local part of the person's whole body, into whole-body motion data, which consists of acceleration data of each local part of the person's whole body. Information processing device. (Note 9) Information processing device, Based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, the first motion calculation unit calculates the first motion features that represent the characteristics of the person's movements. Based on the local motion data, which is motion data for specific parts of a person, the second motion calculation unit calculates a second motion characteristic that represents the characteristics of the person's movements. Based on the teacher motion features, which are pre-identified motion characteristics of a person corresponding to the aforementioned training data, the first motion features, and the second motion features, the parameters used by the second motion calculation unit to calculate the second motion features are updated. Information processing methods. (Note 9.1) The information processing method described in Appendix 9, The difference between the first motion feature and the second motion feature, and the difference between the teacher motion feature and the second motion feature are calculated. The parameters are updated based on each of the calculated differences. Information processing methods. (Appendix 9.2) The information processing method described in Appendix 9.1, As the first motion feature, a first feature quantity representing the motion characteristics of the whole body of a person is calculated based on the whole-body motion data. As the second motion features, based on the local motion data, a second feature quantity representing the motion characteristics of a local part of the person and a second predicted motion representing the predicted motion of the person are calculated. The difference between the first feature and the second feature, and the difference between the teacher action, which represents the identified action of the person that is the teacher action feature, and the second predicted action are calculated. Information processing methods. (Appendix 9.3) The information processing method described in Appendix 9.1, As the first motion feature, a first predicted motion is calculated that represents the motion predicted by the person based on the whole-body motion data. As the second motion feature, a second predicted motion is calculated that represents the motion predicted by the person based on the local motion data. The difference between the first predicted action and the second predicted action, and the difference between the teacher action representing the identified action of the person, which is the teacher action feature, and the second predicted action are calculated. Information processing methods. (Note 10) In an information processing device, Based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, the first motion calculation unit calculates the first motion features that represent the characteristics of the person's movements. Based on the local motion data, which is motion data for specific parts of a person, the second motion calculation unit calculates a second motion characteristic that represents the characteristics of the person's movements. Based on the teacher motion features, which are pre-identified motion characteristics of a person corresponding to the aforementioned training data, the first motion features, and the second motion features, the parameters used by the second motion calculation unit to calculate the second motion features are updated. A program that executes a process. [Explanation of Symbols]

[0068] 10 Information Processing Devices 11 Teacher Modality Motion Recognition Unit 12 Target Modality Motion Recognition Unit 13 Difference calculation part 14 Parameter update section 15 Modality conversion unit 16 Whole-body motion feature extraction unit 17. Local motion feature extraction unit 18 Local motion prediction unit 19 Whole-body motion prediction unit 100 Information Processing Devices 101 CPU 102 ROM 103 RAM 104 Program Groups 105 Storage device 106 Drive unit 107 Communication Interface 108 Input / Output Interfaces 109 Bus 110 Storage medium 111 Communication Network 121 First motion calculation section 122 Second motion calculation section 123 Update Department

Claims

1. A first motion calculation unit calculates a first motion characteristic that represents the characteristics of a person's movements based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, A second motion calculation unit calculates a second motion characteristic that represents the characteristics of a person's movements based on local motion data, which is motion data of a specific part of a person's body. An update unit updates the parameters used by the second motion calculation unit to calculate the second motion characteristics, based on the teacher motion characteristics, which are pre-identified motion characteristics of a person corresponding to the teacher data, the first motion characteristics, and the second motion characteristics. Equipped with an information processing device.

2. An information processing apparatus according to claim 1, The system includes a difference calculation unit that calculates the difference between the first motion feature and the second motion feature, and the difference between the teacher motion feature and the second motion feature. The update unit updates the parameters based on each of the calculated differences. Information processing device.

3. An information processing apparatus according to claim 2, The update unit updates the parameters so that the sum of the calculated differences becomes smaller. Information processing device.

4. An information processing apparatus according to claim 2, The first motion calculation unit calculates a first feature quantity that represents the characteristic quantity of motion in the whole body of a person based on the whole body motion data, as the first motion feature. The second motion calculation unit calculates, as the second motion features, a second feature quantity representing the motion characteristics of a local part of the person, and a second predicted motion representing the motion predicted by the person, based on the local motion data. The difference calculation unit calculates the difference between the first feature and the second feature, and the difference between the teacher action, which represents the identified action of a person that is a teacher action feature, and the second predicted action. Information processing device.

5. An information processing apparatus according to claim 4, The first motion calculation unit and the second motion calculation unit calculate the first feature quantity and the second feature quantity in the same data format. The second motion calculation unit calculates the second predicted motion in the same data format as the teacher motion. Information processing device.

6. An information processing apparatus according to claim 2, The first motion calculation unit calculates a first predicted motion, which represents the motion predicted by the person based on the whole-body motion data, as the first motion feature. The second motion calculation unit calculates a second predicted motion, which represents the motion predicted by the person based on the local motion data, as the second motion feature. The difference calculation unit calculates the difference between the first predicted motion and the second predicted motion, and the difference between the teacher motion, which represents the identified motion of the person that is a teacher motion feature, and the second predicted motion. Information processing device.

7. An information processing apparatus according to claim 6, The first motion calculation unit and the second motion calculation unit calculate the first predicted motion and the second predicted motion in the same data format as the teacher motion. Information processing device.

8. An information processing apparatus according to claim 1, The first motion calculation unit includes a conversion unit that converts the training data, which consists of coordinate data of each local part of the person's whole body, into whole-body motion data, which consists of acceleration data of each local part of the person's whole body. Information processing device.

9. Information processing device, Based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, the first motion calculation unit calculates the first motion features that represent the characteristics of the person's movements. Based on the local motion data, which is motion data for specific parts of a person, the second motion calculation unit calculates a second motion characteristic that represents the characteristics of the person's movements. Based on the teacher motion features, which are pre-identified motion characteristics of a person corresponding to the aforementioned training data, the first motion features, and the second motion features, the parameters used by the second motion calculation unit to calculate the second motion features are updated. Information processing methods.

10. In an information processing device, Based on whole-body motion data, which is motion data of the entire body of a person based on training data representing the movements of a person, the first motion calculation unit calculates the first motion features that represent the characteristics of the person's movements. Based on the local motion data, which is motion data for specific parts of a person, the second motion calculation unit calculates a second motion characteristic that represents the characteristics of the person's movements. Based on the teacher motion features, which are pre-identified motion characteristics of a person corresponding to the aforementioned training data, the first motion features, and the second motion features, the parameters used by the second motion calculation unit to calculate the second motion features are updated. A program that executes a process.

Images