Training device, evaluation device, training method, evaluation method, and program

The integration of motion video and sensor data in a learning device improves the accuracy of motion evaluation by creating a model that accounts for environmental variations, enhancing scoring reliability in diverse conditions.

WO2024247080A1PCT designated stage expired Publication Date: 2024-12-05NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/020016
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing computer-based evaluation systems for motion, such as in figure skating, surfing, and medical procedures, face accuracy issues due to environmental variations like lighting and weather, affecting scoring even when the motion remains the same.

Method used

A learning device and method that combines motion video data with sensor data from devices used during the motion, creating a mathematical model to evaluate motion based on both video and measurement data, reducing environmental impact on evaluation accuracy.

Benefits of technology

This approach enhances the accuracy of motion evaluation by using sensor data that is less affected by environmental factors, providing a more reliable assessment compared to evaluations based solely on video data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023020016_05122024_PF_FP_ABST
    Figure JP2023020016_05122024_PF_FP_ABST
Patent Text Reader

Abstract

One aspect of the present invention is a training device comprising a control unit that trains a training target by using: action video data that is video data of a video showing an action of an evaluation target, which is a person or animal whose action is to be evaluated; action measurement data indicating time-series results, which are obtained during an action shown in the action video data, of an action measurement sensor that is a sensor attached to at least one of the body of the evaluation target, an item worn by the evaluation target, and an item used by the evaluation target during the action, and obtains a result according to the action of the evaluation target; and correct data indicating a result of evaluating the action. The training target is a mathematical model that evaluates the action on the basis of the action video data and the action measurement data.
Need to check novelty before this filing date? Find Prior Art

Description

Learning device, evaluation device, learning method, evaluation method, and program

[0001] The present invention relates to a learning device, an evaluation device, a learning method, an evaluation method, and a program.

[0002] There is growing interest in technology that allows computers to evaluate the movements of people being evaluated based on video footage, such as in scoring competitions such as figure skating, surfing, and diving, and in scoring the movements of medical trainees during medical procedures such as surgery.

[0003] Paritosh Parmar and Brendan Tran Morri, “What and How Well You Performed? A Multitask Learning Approach to Action Quality Assess”, arXiv:1904.04346v2, (2019)

[0004] However, the environments in which sports and medical procedures are performed are not always the same. For example, the lighting conditions may be different for indoor movements, and the weather conditions may be different for outdoor movements. The video captures not only the subject being evaluated, but also these environmental differences. As a result, even if the subject's movements are the same, the computer evaluation results may differ. In other words, the accuracy of the computer evaluation may be poor. This situation is not limited to cases where the subject is a human, but is also the same when the subject is an animal, such as a dog or a cat.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for improving the accuracy of motion evaluation.

[0006] One aspect of the present invention is a learning device that includes: motion video data, which is video data of a video showing the motion of an evaluation subject, which is a human or animal whose motion is to be evaluated; motion measurement data, which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor, which is a sensor attached to at least one of the body of the evaluation subject, an item worn by the evaluation subject, and an item used by the evaluation subject when performing the motion, and which obtains results corresponding to the motion of the evaluation subject; and ground truth data, which indicates the result of evaluation of the motion; and a control unit that learns a learning subject, wherein the learning subject is a mathematical model that evaluates the motion based on the motion video data and the motion measurement data.

[0007] One aspect of the present invention is an evaluation device comprising: an interface unit that acquires a pair of action video data showing the action of an evaluation subject and action measurement data obtained during the action shown in the action video data; a control unit that learns a learning subject using the action video data, which is video data of an image showing the action of an evaluation subject, which is a person or animal whose action is to be evaluated; action measurement data, which is a time series of results obtained during the action shown in the action video data by a action measurement sensor, which is a sensor attached to at least one of the body of the evaluation subject, an item worn by the evaluation subject, and an item used by the evaluation subject when performing the action, and which obtains a result corresponding to the action of the evaluation subject; and ground truth data that indicates the result of evaluation of the action, wherein the learning subject is a mathematical model that evaluates the action based on the action video data and the action measurement data; and a trained evaluation model execution unit that uses the trained mathematical model obtained by the learning device and the pair acquired by the interface unit to evaluate the action shown in the action video data included in the pair.

[0008] One aspect of the present invention is a learning method comprising a control step of learning a learning object using action video data, which is video data of an image showing the action of an evaluation object, which is a human or animal whose action is to be evaluated, action measurement data, which is a time series of results obtained by a action measurement sensor, which is a sensor attached to at least one of the body of the evaluation object, an item worn by the evaluation object, and an item used by the evaluation object when performing the action, during the action shown in the action video data, during the action shown in the action video data, and ground truth data indicating the result of evaluation of the action, wherein the learning object is a mathematical model that evaluates the action based on the action video data and the action measurement data.

[0009] One aspect of the present invention is an evaluation method comprising: an interface step of acquiring a pair of action video data showing the action of an evaluation subject and action measurement data obtained during the action shown in the action video data; a control step of performing learning of a learning subject using the action video data, which is video data of an image showing the action of an evaluation subject, which is a human or animal whose action is to be evaluated; action measurement data, which is a time series of results obtained during the action shown in the action video data by a action measurement sensor, which is a sensor attached to at least one of the body of the evaluation subject, an item worn by the evaluation subject, and an item used by the evaluation subject when performing the action, and which obtains a result corresponding to the action of the evaluation subject, and ground truth data indicating the result of evaluation of the action; wherein the learning subject is a mathematical model that evaluates the action based on the action video data and the action measurement data; and a learned evaluation model execution step of evaluating the action shown in the action video data included in the set using the learned mathematical model obtained by the learning method and the set obtained in the interface step.

[0010] One aspect of the present invention is a program for causing a computer to function as either or both of the learning device and the evaluation device.

[0011] The present invention makes it possible to provide a technique for improving the accuracy of motion evaluation.

[0012] A diagram showing an example of the configuration of an evaluation system according to an embodiment. A diagram showing an example of the hardware configuration of a learning device according to an embodiment. A diagram showing an example of the hardware configuration of an evaluation device according to an embodiment. A flowchart showing an example of the flow of processing executed by a learning device according to an embodiment. A flowchart showing an example of the flow of processing executed by an evaluation device according to an embodiment. A diagram showing an example of a first modified processing according to a modified example. An explanatory diagram explaining an example of the processing of assigning a focus identifier according to a modified example.

[0013] 1 is a diagram showing an example of the configuration of an evaluation system 100 according to an embodiment. The evaluation system 100 includes a learning apparatus 1 and an evaluation apparatus 2. The learning apparatus 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected via a bus, and executes a program.

[0014] The control unit 11 executes a learning process. The learning process is a process for learning a mathematical model of a learning object. The mathematical model of a learning object is a mathematical model (hereinafter referred to as an "evaluation model") that evaluates the movement of an evaluation object using a set of movement video data, movement measurement data, and ground truth data (hereinafter referred to as "learning data").

[0015] The action video data is video data of an image showing the action of a person or animal (hereinafter referred to as "evaluation subject") whose action is to be evaluated.

[0016] The evaluation target may be any person or animal as long as the motion is an evaluation target. The evaluation target may be, for example, an athlete in figure skating, surfing, diving, or the like. When the evaluation target is an athlete, the motion evaluated by the evaluation model is the athlete's motion during the competition. The evaluation target may be, for example, a medical intern. When the evaluation target is a medical intern, the motion evaluated by the evaluation model is the motion during a medical procedure such as surgery.

[0017] The motion measurement data is a time series of results obtained by a motion measurement sensor during the motion captured in the motion video data. The motion measurement sensor is a sensor attached to at least one of the body of the subject to be evaluated, an item worn by the subject to be evaluated, and an item used by the subject to be evaluated during the motion, and obtains results corresponding to the motion of the subject to be evaluated.

[0018] The motion measurement sensor is, for example, an acceleration sensor. In such a case, the result obtained by the motion measurement sensor is acceleration. Therefore, in such a case, the time series of the results obtained by the motion measurement sensor is a time series of acceleration. The motion measurement sensor is, for example, an angular velocity sensor. In such a case, the result obtained by the motion measurement sensor is angular velocity. Therefore, in such a case, the time series of the results obtained by the motion measurement sensor is a time series of angular velocity.

[0019] Alternatively, the motion measurement sensor may be, for example, an inertial measurement unit (IMU), or a sensor that obtains biological signals that indicate changes in response to the motion of the subject being evaluated, such as heart rate, brain waves, pulse, blood pressure, breathing, and sweating.

[0020] If a sensor that obtains results according to the motion of the evaluation subject is attached to the body of the evaluation subject or to something worn by the evaluation subject, the results obtained by the sensor will be less affected by the environment, such as the intensity and color of lighting, than images, and will have a high correlation with the motion of the evaluation subject. Therefore, evaluation based on not only motion video data but also motion measurement data will be more accurate than evaluation based only on motion video data.

[0021] This section explains what the subject of evaluation uses when making movements. For example, if the sport is surfing, the subject of evaluation uses a surfboard during the competition. The surfboard moves in many scenes during the competition in a way that is highly correlated with the movements of the subject of evaluation. Therefore, if a sensor is attached to the surfboard, results that are highly correlated with the movements of the subject of evaluation can be obtained, even if the sensor is not attached to the body of the subject of evaluation. Again, this result is less affected by environmental factors such as lighting intensity and color than video.

[0022] Therefore, even if the sensor is attached to something used during the movement of the evaluation subject, using the time series of results obtained by the sensor will result in a more accurate evaluation than an evaluation based solely on movement video data. This situation is not limited to surfing, but is common to events where movements using equipment such as skis and snowboards are the subject of scoring.

[0023] Furthermore, when the motion evaluated by the evaluation model is a medical procedure, the movement of a medical device used during the procedure, such as a scalpel, also has a high correlation with the motion of the measurement subject. Therefore, even if the motion is during a medical procedure, if a motion measurement sensor is attached to the medical device, the motion measurement sensor will obtain results that are highly correlated with the motion of the evaluation subject, even if the motion measurement sensor is not attached to the body of the evaluation subject. This result is also less affected by environmental factors such as lighting intensity and color than video.

[0024] Therefore, even if the movement being evaluated is a medical procedure and the sensor is attached to something used during the movement of the subject being evaluated, using the time series of results obtained by the sensor can provide a more accurate evaluation than an evaluation based solely on movement video data.

[0025] In addition, if the subject to be evaluated is an animal such as a dog or cat, the motion measurement sensor may be attached to, for example, a collar, or if a tag is attached to the foot or the like, it may be attached to the tag.

[0026] Such a motion measurement sensor may be a sensor that obtains information on the movement of the object to be evaluated in each of three linearly independent directions, such as forward / backward, left / right, and up / down, or may be a sensor that obtains information in two or one dimensions.

[0027] Note that the motion measurement data used by the evaluation model does not have to be the results obtained by a single motion measurement sensor. The evaluation model may use motion measurement data obtained by multiple motion measurement sensors. In such a case, multiple motion measurement data are input to the evaluation model. The multiple motion measurement sensors may be multiple types of motion measurement sensors that obtain different physical quantities, such as a motion measurement sensor that obtains acceleration and a motion measurement sensor that obtains biological information. In such a case, the evaluation model uses time series of multiple different types of physical quantities as motion measurement data for evaluation.

[0028] The correct answer data is data indicating the result of evaluation of the motion of the evaluation target.

[0029] In the learning process, the control unit 11 updates the evaluation model based on the evaluation result of the evaluation model and the correct answer data so as to reduce the difference between the evaluation result of the evaluation model and the correct answer data. The difference may be expressed, for example, by the L1 norm, the L2 norm, or the linear sum of the L1 norm and the L2 norm.

[0030] The learning process is executed until a predetermined termination condition for learning (hereinafter referred to as the "learning termination condition") is satisfied. The learning termination condition is, for example, a condition that the change in the learning object due to learning is smaller than a predetermined change. The learning termination condition may also be, for example, a condition that the learning object has been updated a predetermined number of times or more.

[0031] The evaluation device 2 evaluates the movement of the evaluation target using a trained evaluation model, which is the evaluation model at the time when the learning termination condition is satisfied. The evaluation device 2 accepts input of a set of movement video data showing the movement of the evaluation target and movement measurement data obtained during the movement shown in the movement video data (hereinafter referred to as "inference stage data"). The evaluation device 2 performs evaluation based on the accepted inference stage data.

[0032] <Regarding the Evaluation Model> The evaluation model may be of any type as long as it can evaluate the motion of the evaluation target using motion video data, motion measurement data, and ground truth data.

[0033] The evaluation model includes, for example, a video data feature amount acquisition process and a measurement data feature amount acquisition process. The video data feature amount acquisition process is a process for acquiring feature amounts of motion video data. The measurement data feature amount acquisition process is a process for acquiring feature amounts of motion measurement data.

[0034] The evaluation model including the video data feature amount acquisition process and the measurement data feature amount acquisition process evaluates the motion of the evaluation object based on the results of the video data feature amount acquisition process and the results of the measurement data feature amount acquisition process.

[0035] <Regarding processing targets for feature acquisition> In the video data feature acquisition process, feature amounts may be obtained for each of the divided groups (hereinafter referred to as "partial video data") of motion video data that has been divided into a plurality of groups along the time axis. Therefore, in this case, feature amounts for each partial video data are obtained.

[0036] For example, if the video is motion video data consisting of P x Q frames (P is an integer equal to or greater than 1, and Q is an integer equal to or greater than 2), the video feature acquisition process divides the video into partial video data for each P frames in frame number order, and acquires features for each partial video data. Therefore, in such a case, the video data feature acquisition process obtains Q features from the motion video data.

[0037] In addition, when the process of dividing the motion video data into partial video data (hereinafter referred to as the "video data division process") is performed, it may be performed at any timing as long as it is before the video data feature acquisition process is performed.

[0038] The video data division process is a process of dividing the motion video data along the time axis for each predetermined number of frames, for example.

[0039] Note that motion video data divided into partial video data is also a type of motion video data. Therefore, the motion video data input to the evaluation model may be motion video data after video data division processing. When the motion video data input to the evaluation model is motion video data after video data division processing, the video data division processing may be performed, for example, by the control unit 11 or by a device other than the learning device 1.

[0040] When another device executes the evaluation, the control unit 11 acquires the result, for example, via the interface unit 12 described later, and uses it for evaluation by executing the evaluation model. Note that the operation video data input to the evaluation model may be operation video data before the video data division process, and in such a case, when the video data division process is also executed, the evaluation model executes the video data division process.

[0041] The same applies to the measurement data feature amount acquisition process. In the measurement data feature amount acquisition process, feature amounts may be obtained for each group (hereinafter referred to as "partial measurement data") of the motion measurement data that has been divided into a plurality of groups at a predetermined time interval along the time axis. Therefore, in this case, feature amounts for each partial measurement data are obtained.

[0042] When feature amounts are obtained for each of a plurality of partial measurement data of the motion measurement data, the number of feature amounts is greater than that obtained when the motion measurement data is not divided into partial measurement data.

[0043] In addition, when the process of dividing the motion measurement data into partial measurement data (hereinafter referred to as the "measurement data division process") is performed, it may be performed at any timing as long as it is before the measurement data feature acquisition process is performed.

[0044] The measurement data division process is a process of dividing measurement video data, which is a type of time series, along a time axis for each predetermined time width, for example.

[0045] Note that motion measurement data that has been divided into partial measurement data is also a type of motion measurement data. Therefore, the motion measurement data input to the evaluation model may be motion measurement data after measurement data division processing. When the motion measurement data input to the evaluation model is motion measurement data after measurement data division processing, the measurement data division processing may be executed by, for example, the control unit 11 or by a device other than the learning device 1.

[0046] When another device executes the evaluation, the control unit 11 acquires the result, for example, via the interface unit 12 described later, and uses it for evaluation by executing the evaluation model. Note that the motion measurement data input to the evaluation model may be the motion measurement data before the measurement data division process, and in such a case, when the measurement data division process is also executed, the evaluation model executes the measurement data division process.

[0047] <Regarding how features are used for evaluation> As described above, the evaluation model evaluates the motion of the evaluation object based on the results of the video data feature acquisition process and the results of the measurement data feature acquisition process. In evaluating the motion of the evaluation object based on the results of the video data feature acquisition process and the results of the measurement data feature acquisition process, the results of the video data feature acquisition process and the measurement data feature acquisition process may be used in an integrated state to evaluate the motion of the evaluation object.

[0048] When evaluating the movement of the evaluation object based on the results of the video data feature acquisition process and the results of the measurement data feature acquisition process, the results of the video data feature acquisition process and the results of the measurement data feature acquisition process may be used to evaluate the movement of the evaluation object without being integrated.

[0049] Integration is a process of combining different data into one, such as obtaining the direct product of vectors when the data is expressed as vectors, or concatenating matrices in the row or column direction when the data is expressed as matrices.

[0050] For the integration, one or more fully connected layers may be used, or the Transformer Encoder described in the following Reference 1 may be used. In this case, the number of Transformer Blocks may be any number, and for example, two Transformer Blocks may be connected in succession.

[0051] Reference 1: Vaswani, Ashish, et al. “Attention is all you need.” Advances in neural information processing systems 30 (2017).

[0052] 2 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. As described above, the learning device 1 includes a control unit 11 and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0053] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0054] The control unit 11 performs, for example, a learning process. The control unit 11 performs, for example, a video data division process. The control unit 11 performs, for example, a measurement data division process. The control unit 11 controls, for example, the operation of each functional unit provided in the learning device 1. The control unit 11 acquires learning data, for example, via the interface unit 12. The control unit 11 acquires, for example, information stored in the memory unit 13. The process of acquiring information stored in the memory unit 13 is specifically reading.

[0055] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits learning data. In such a case, the interface unit 12 acquires learning data by communicating with the device that transmits the learning data.

[0056] When an external device performs video data division processing, the interface unit 12 obtains the motion video data after the video data division processing by the external device as motion video data including learning data. When an external device performs measurement data division processing, the interface unit 12 obtains the motion measurement data after the measurement data division processing by the external device as motion measurement data including learning data.

[0057] The external device is, for example, the evaluation device 2. The interface unit 12 transmits the trained evaluation model obtained by the learning process (i.e., the trained evaluation model) to the evaluation device 2 through communication with the evaluation device 2.

[0058] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the interface unit 12 accepts input of various information to the learning device 1 via the input device, either wired or wireless. Note that the learning data does not necessarily have to be input to a communication interface, but may also be input to an input device.

[0059] The interface unit 12 may output various types of information. The interface unit 12 may include a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the learning device 1. The interface unit 12 outputs, for example, information input to the interface unit 12.

[0060] The memory unit 13 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The memory unit 13 stores various information related to the learning device 1. The memory unit 13 stores, for example, information necessary for the learning process. The memory unit 13 stores, for example, an evaluation model in advance. The memory unit 13 stores, for example, various information generated by the operation of the control unit 11. Therefore, the memory unit 13 stores, for example, a trained evaluation model. The memory unit 13 stores, for example, information acquired by the interface unit 12.

[0061] 3 is a diagram illustrating an example of the hardware configuration of the evaluation device 2 according to the embodiment. As described above, the evaluation device 2 includes the control unit 21 (an example of a trained evaluation model execution unit) and executes a program. By executing the program, the evaluation device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0062] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, whereby the evaluation device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0063] The control unit 21 executes the trained evaluation model, for example, using the inference stage data as the execution target. As a result, the control unit 21 evaluates the actions depicted in the action video data included in the inference stage data. That is, the control unit 21 uses the trained evaluation model and the inference stage data to evaluate the actions depicted in the action video data included in the inference stage data.

[0064] If the control unit 11 has executed the video data division process during the learning stage of the evaluation model but the video data division process is not included in the evaluation model, the control unit 21 executes the video data division process before executing the learned evaluation model. If the control unit 11 has executed the measurement data division process during the learning stage of the evaluation model but the measurement data division process is not included in the evaluation model, the control unit 21 executes the measurement data division process before executing the learned evaluation model.

[0065] The control unit 21 controls, for example, the operation of each functional unit included in the evaluation device 2. The control unit 21 acquires, for example, various pieces of information acquired via the interface unit 22. Therefore, for example, the control unit 21 acquires data (i.e., inference stage data) on which the trained evaluation model is to be executed. The control unit 21 acquires, for example, information stored in the memory unit 23. Specifically, the process of acquiring information stored in the memory unit 23 is reading.

[0066] The interface unit 22 includes a communication interface for connecting the evaluation device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, a device that has transmitted the inference stage data. In such a case, the interface unit 22 acquires the inference stage data by communicating with the device that has transmitted the inference stage data. The external device is, for example, the learning device 1. The interface unit 22 acquires the trained evaluation model by communicating with the learning device 1.

[0067] When an external device executes the video data division process, the interface unit 22 obtains the motion video data after the video data division process by the external device as motion video data including the inference stage data. When the external device executes the video data division process in the learning stage of the evaluation model, the evaluation device 2 also obtains the result of the video data division process executed by the external device.

[0068] When an external device executes the measurement data division process, the interface unit 22 obtains the motion measurement data after the measurement data division process by the external device as motion measurement data including the inference stage data. When the external device executes the measurement data division process in the learning stage of the evaluation model, the evaluation device 2 also obtains the result of the measurement data division process executed by the external device.

[0069] The interface unit 22 includes input devices such as a mouse, keyboard, and touch panel. The interface unit 22 may be configured as an interface that connects these input devices to the evaluation device 2. In this way, the interface unit 22 accepts input of various information to the evaluation device 2 via the input device, either wired or wireless. Note that the inference stage data does not necessarily have to be input to a communication interface, but may be input to an input device.

[0070] The interface unit 22 outputs various types of information. The interface unit 22 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The interface unit 22 may be configured as an interface that connects these display devices to the evaluation device 2. The interface unit 22 outputs, for example, information input to the interface unit 22.

[0071] The storage unit 23 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the evaluation device 2. The storage unit 23 stores, for example, a trained evaluation model. The storage unit 23 stores, for example, various information generated by the operation of the control unit 21. The storage unit 23 stores, for example, information acquired by the interface unit 22.

[0072] 4 is a flowchart showing an example of the flow of processing executed by the learning device 1 in this embodiment. The control unit 11 acquires learning data (step S101). The control unit 11 executes a learning process using the acquired learning data until a learning termination condition is met (step S102). The control unit 11 outputs the trained evaluation model to a predetermined output destination (step S103). The predetermined output destination is, for example, the storage unit 13. Note that output to the storage unit 13 means writing to the storage unit 13. The predetermined output destination may be, for example, an external device such as the evaluation device 2 connected to be able to communicate via the interface unit 12.

[0073] 5 is a flowchart showing an example of the flow of processing executed by the evaluation device 2 in the embodiment. For ease of explanation, it is assumed that the trained evaluation model has been stored in advance in the storage unit 23 in the example of the flowchart in FIG.

[0074] Inference stage data is input to the interface unit 22, and the control unit 21 acquires the inference stage data input to the interface unit 22 (step S201). Inputting the inference stage data to the interface unit 22 means that the interface unit 22 acquires the inference stage data.

[0075] The control unit 21 executes the learned evaluation model on the inference stage data obtained in step S201 (step S202). By the processing of step S202, the actions depicted in the action video data included in the inference stage data obtained in step S201 are evaluated. After step S202, the control unit 21 outputs the evaluation result obtained in step S202 to a predetermined output destination (step S203). The predetermined output destination is, for example, the storage unit 23. Note that output to the storage unit 23 means writing to the storage unit 23. The predetermined output destination may be, for example, a predetermined external device connected so as to be able to communicate via the interface unit 22.

[0076] The learning device 1 configured in this manner learns an evaluation model based not only on motion video data but also on motion measurement data. Because the motion measurement data is obtained by a motion measurement sensor, it is less affected by environmental factors such as lighting intensity and color than video. Therefore, an evaluation model trained based on not only motion video data but also motion measurement data can provide evaluations with higher accuracy than those based solely on motion video data. Therefore, the learning device 1 can improve the accuracy of motion evaluations.

[0077] Furthermore, the evaluation device 2 configured in this manner evaluates the movement using the learned mathematical model obtained by the learning device 1. Therefore, the evaluation device 2 can improve the accuracy of the evaluation of the movement.

[0078] Furthermore, the evaluation system 100 configured in this manner includes the learning device 1. Therefore, the evaluation system 100 can improve the accuracy of the evaluation of actions.

[0079] (Modification) In addition, when evaluating the movement of an evaluation object based on the feature amounts obtained in the video data feature amount acquisition process and the feature amounts obtained in the measurement data feature amount acquisition process, for example, the following first transformation process may be executed. The first transformation process is a process in which the dimensions of the feature amounts obtained in the video data feature amount acquisition process and the feature amounts obtained in the measurement data feature amount acquisition process are aligned via a linear layer, input into a transformer block via embedding processing, and evaluated through an estimator. The estimator is, for example, a plurality of fully connected layers.

[0080] In the first transformation process, the linear layer processing is a process for aligning the dimensions of the feature amounts obtained in the video data feature acquisition process and the feature amounts obtained in the measurement data feature acquisition process, and therefore does not necessarily have to be executed if the dimensions of the feature amounts obtained in the video data feature acquisition process and the feature amounts obtained in the measurement data feature acquisition process are originally the same. Furthermore, the linear layer may have any structure, and may be, for example, a single fully connected layer.

[0081] We will now explain the embedding process. Because the Transformer block handles inputs in parallel, it cannot take into account the chronological order of the inputs or whether the input is the result of video data feature acquisition processing or measurement data feature acquisition processing. "Not being able to take into account" means that such information cannot be used. Therefore, when using the Transformer, information indicating the attributes of the features input to the Transformer must be provided to the Transformer block, which is the layer that executes the Transformer, through the embedding process.

[0082] The information indicating the attribute of the feature includes, for example, information indicating whether the feature is the result of a video data feature acquisition process or a measurement data feature acquisition process. If the feature is the result of a video data feature acquisition process, the information indicating the attribute may include information indicating which partial video data the feature is a feature of. If the feature is the result of a measurement data feature acquisition process, the information indicating the attribute may include information indicating which partial measurement data the feature is a feature of.

[0083] Therefore, the embedding process acquires information indicating the attributes of the feature quantities used in the transformer so that the transformer process is performed based on information indicating the attributes of the feature quantities.

[0084] The embedding process in the first transformation process is, for example, sensor embedding. Sensor embedding is a process of assigning an identifier to each feature input to the Transformer, identifying whether it is the result of a video data feature acquisition process or a measurement data feature acquisition process. The identifier may be any identifier as long as it can identify whether it is the result of a video data feature acquisition process or a measurement data feature acquisition process.

[0085] Therefore, the identifier may be, for example, "1" to indicate the result of the video data feature acquisition process and "0" to indicate the result of the measurement data feature acquisition process. Also, instead of integers such as 1 or 0, it may be a decimal point such as 0.1, or it may not even be a number.

[0086] The identifier to be assigned is determined in advance depending on the result of the video data feature amount acquisition process and the result of the measurement data feature amount acquisition process.

[0087] The embedding process in the first transformation process is, for example, positional embedding. Positional embedding is a process in which, if the feature is the result of a video data feature acquisition process, an identifier is assigned to indicate which partial video data the feature is from, and, if the feature is the result of a measurement data feature acquisition process, an identifier is assigned to indicate which partial measurement data the feature is from. The assigned identifier is, for example, an identifier that indicates the temporal order of the partial video data or partial measurement data.

[0088] FIG. 6 is a diagram showing an example of the first transformation process in the modified example. In the example of FIG. 6, one piece of motion video data and two pieces of motion measurement data are used. In the example of FIG. 6, one piece of motion video data is divided into partial video data, and both pieces of motion measurement data are divided into partial measurement data. In the example of FIG. 6, both the partial video data and the partial measurement data are expressed as "Clip." In the example of FIG. 6, Position Embedding and Sensor Embedding are performed. In the example of FIG. 6, a Linear layer is provided, and the output of the Linear layer, together with the results of Position Embedding and Sensor Embedding, is input to a Transformer block. In the example of FIG. 6, evaluation is performed based on the results of the Transformer block.

[0089] <Technology enabling non-homogeneous data processing> In the explanation so far, the importance of each sample in the time series of motion video data and motion measurement data has not been explicitly assigned when evaluating motion. In other words, all samples have been treated uniformly for evaluation.

[0090] However, video image data and motion measurement data may contain motions that are not necessarily suitable for evaluating motion. For example, in a judged sport, this data may include data from the period before the competition begins or the period after the competition when players are substituted. When evaluating motion, the motions indicated by this data are not subject to evaluation, and may therefore act as noise when evaluating motion.

[0091] Therefore, the learning device 1 may suppress the occurrence of such a situation by making the time-series samples in the action video data and the action measurement data non-uniform. Specifically, the evaluation model performs evaluation based on a focus identifier, which is an identifier that indicates that the influence on the action evaluation is stronger than others in the action video data and the action measurement data. In such a case, the evaluation model can perform evaluation with higher accuracy.

[0092] The attention identifier is assigned to samples in a period that includes the time when a predetermined index indicating the degree of intensity of the movement is highest in the graph of the movement measurement data and that satisfies a predetermined condition. The evaluation model weights samples that have been assigned an attention identifier more heavily than samples that have not been assigned an attention identifier.

[0093] The predetermined condition may be any condition that specifies a period that includes the time when a predetermined index indicating the degree of intensity of a movement is highest in the graph shown by the movement measurement data, for example, a condition that specifies each time width before and after the time when a predetermined index indicating the degree of intensity of a movement is highest in the graph shown by the movement measurement data as the predetermined time width.

[0094] As described above, the evaluation model weights samples with a notable identifier more heavily than samples without it. Therefore, the more intense the movement, the more strongly it should affect the evaluation of the movement. Therefore, the notable identifier is assigned to samples from a period that includes the time when a predetermined index indicating the degree of movement intensity is highest in the graph shown by the movement measurement data and that satisfies predetermined conditions. As a result, the evaluation model can perform evaluation with higher accuracy.

[0095] In addition, assigning a target identifier to data means determining that the data to be judged satisfies the content indicated by the target identifier, and recording information indicating the judgment result in a specified storage device from which an evaluation model such as storage unit 13 or storage unit 23 can read information.

[0096] In addition, the process of making the time series samples in the action video data and the action measurement data non-uniform includes a process of dividing the action video data and the action measurement data into multiple Cilps, and in the process of dividing the clips, the action video data and the action measurement data may be divided into multiple Cilps so that the number of clips included in a period that includes the time when a predetermined index indicating the degree of intensity of the action is highest in the graph shown by the action measurement data and that satisfies predetermined conditions is greater than in other periods.

[0097] Fig. 7 is an explanatory diagram illustrating an example of a process for assigning an attention identifier in a modified example. Fig. 7 shows three types of time series. The three types of time series are IMU signals. In the example of Fig. 7, an attention identifier is assigned to motion video data and motion measurement data within a period that satisfies predetermined attention period conditions that at least include a condition that the maximum differential coefficient time is included within the period.

[0098] The maximum derivative time is the time at which the derivative is at its maximum value in the graph of the motion measurement data. When a subject is moving, the value of the motion measurement data should change more drastically than when the subject is not moving.

[0099] Therefore, if a target identifier is assigned so as to include the time at which the maximum differential coefficient is obtained in the graph of the movement measurement data, the sample to which the target identifier is assigned is likely to be data obtained while the movement is being performed. Therefore, in the example of Figure 7, the time at which the maximum differential coefficient is obtained is an example of the time at which the predetermined index indicating the degree of intensity of the movement is highest in the graph shown by the movement measurement data.

[0100] In the example of Figure 7, the set of samples assigned with a noteworthy identifier are samples from the period indicated as "Technique". In Figure 7, the horizontal axis represents time. In Figure 7, "Preparation Period" is the period when the athlete is preparing. In Figure 7, "Technique" is the period when the athlete is performing an action. In Figure 7, "Post-Execution" is the period when the athlete has finished performing an action and is not performing an action. In Figure 7, time t = ta is an example of the maximum derivative time.

[0101] Hereinafter, the process of assigning an attention identifier to the action video data and the action measurement data within a period that satisfies the attention period condition based on the action measurement data will be referred to as attention identifier assignment process.

[0102] The attention period condition may be any condition including a condition that the maximum differential coefficient time is included within the period. The attention period condition may be, for example, a condition that the period has a predetermined time width before and after the maximum differential coefficient time. The attention period condition may be, for example, a condition that the period has a predetermined time width before and after the maximum differential coefficient time and is longer than a period that does not satisfy the attention period condition.

[0103] The condition for the period of interest may be, for example, a period having a predetermined time width before and after the maximum differential coefficient time, and the length of the period is a predetermined length that is at least 1 / 3 of the time from the start to the end of the video shown by the motion video data.

[0104] The meaning of "1 / 3" will be explained. The video shown by the motion video data may show a preparation period for the athlete, a period during which the athlete is preparing, a period during which the athlete is performing, and a period after the athlete has finished performing. In such a case, if the preparation period for the athlete, the period during which the athlete is preparing, the period during which the athlete is performing, and the period after the athlete has finished performing are all of the same duration, it is preferable to assign to the evaluation model a period centered on the time of the maximum differential coefficient and having a length of 1 / 3 of the length of the video shown by the motion video data.

[0105] However, when evaluating a movement, it is expected that the user will delete unnecessary data and input the deleted data as movement video data into the learning device 1. In such a case, the attention period condition is preferably, for example, a period having a predetermined time width before and after the maximum differential coefficient time, and the length of the period is a predetermined length that is at least one-third of the time from the start to the end of the video shown in the movement video data.

[0106] The attention identifier assignment process may be performed at any timing before the evaluation model determines the evaluation of the action. The attention identifier assignment process is performed, for example, before the video data feature amount acquisition process and the measurement data feature amount acquisition process are performed. In such a case, the evaluation model acquires features by weighting data to which an attention identifier has been assigned more heavily than other data in the video data feature amount acquisition process and the measurement data feature amount acquisition process.

[0107] When the attention identifier assignment process is executed before the video data feature amount acquisition process and the measurement data feature amount acquisition process are executed, the attention identifier assignment process may be executed, for example, after the video data division process and the measurement data division process are executed. Note that when the video data division process and the measurement data division process are executed, the attention identifier assignment process may be assigned to each partial video data and partial measurement data, thereby assigning an attention identifier to the sample.

[0108] The attention identifier assignment process may be executed after the video data division process and the measurement data division process, for example, after the video data feature amount acquisition process and the measurement data feature amount acquisition process. In such a case, since attention identifiers and features are obtained for each partial video data and partial measurement data, the evaluation model may determine the evaluation by assigning a large weight to the feature amounts to which attention identifiers are assigned among the obtained feature amounts.

[0109] The attention identifier assignment process may be executed by the control unit 11 during the learning stage, or may be executed by an external device when the video data division process and the measurement data division process are executed by the external device. When executed by an external device, the control unit 11 acquires the results of the attention identifier assignment process together with the division results of the video data division process and the measurement data division process via the interface unit 12.

[0110] It should be noted that when the attention identifier assignment process is executed in the learning stage, the evaluation device 2 also uses the result of the attention identifier assignment process. For example, when the control unit 11 executes the attention identifier assignment process in the learning stage, the control unit 21 acquires the result of the attention identifier assignment process by, for example, executing the attention identifier assignment process by the control unit 21. For example, when the attention identifier assignment process is executed in the learning stage by the control unit 11, the control unit 21 may acquire the result of the attention identifier assignment process by having an external device execute the attention identifier assignment process and acquiring the result.

[0111] For example, when an external device executes the attention identifier assignment process in the learning stage, the control unit 21 acquires the result of the attention identifier assignment process by, for example, executing the attention identifier assignment process by the control unit 21. For example, when an external device executes the attention identifier assignment process in the learning stage, the control unit 21 may acquire the result of the attention identifier assignment process by having the external device execute the attention identifier assignment process and acquiring the result.

[0112] The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0113] The evaluation device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that the devices can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0114] The learning device 1 and the evaluation device 2 do not necessarily need to be packaged in different housings, but may be packaged together in a single housing. In such a case, the learning device 1 and the evaluation device 2 may be a computer that functions as both the learning device 1 and the evaluation device 2 by executing a program.

[0115] Note that all or part of the functions of the learning device 1 and the evaluation device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0116] The control unit 21 is an example of a trained evaluation model execution unit.

[0117] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0118] REFERENCE SIGNS LIST 100...Evaluation system, 1...Learning device, 2...Evaluation device, 11...Control unit, 12...Interface unit, 13...Memory unit, 21...Control unit, 22...Interface unit, 23...Memory unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory

Claims

1. A learning device comprising: a control unit that learns a learning object using motion video data, which is video data of an image showing the motion of an evaluation object which is a human or animal whose motion is to be evaluated; motion measurement data, which is a time series of results obtained during the motion captured in the motion video data by a motion measurement sensor which is a sensor attached to at least one of the body of the evaluation object, an item worn by the evaluation object, and an item used by the evaluation object when performing the motion, and which obtains results according to the motion of the evaluation object, during the motion captured in the motion video data; and ground truth data which indicates the result of the evaluation of the motion; wherein the learning object is a mathematical model which evaluates the motion based on the motion video data and the motion measurement data.

2. The learning device according to claim 1, wherein the mathematical model obtains features of the partial video data for each set of partial video data after the motion video data is divided into a plurality of sets along a time axis.

3. The learning device according to claim 1, wherein the mathematical model obtains features of the partial measurement data for each set of partial measurement data obtained after dividing the motion measurement data into a plurality of sets at a predetermined time width along a time axis.

4. The learning device according to claim 1, wherein the mathematical model performs the evaluation also based on a focus identifier that indicates that an influence on the evaluation of the movement is stronger than others in the movement video data and the movement measurement data.

5. An evaluation device comprising: an interface unit that acquires a set of motion video data showing the motion of an evaluation subject and motion measurement data obtained during the motion shown in the motion video data; a control unit that performs learning of a learning subject using motion video data, which is video data of a video showing the motion of an evaluation subject which is a human or animal whose motion is to be evaluated, motion measurement data which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor which is a sensor attached to at least one of the body of the evaluation subject, an item worn by the evaluation subject, and an item used by the evaluation subject when performing the motion and which obtains a result according to the motion of the evaluation subject, and ground truth data which indicates the result of evaluation of the motion, wherein the learning subject is a mathematical model which evaluates the motion based on the motion video data and the motion measurement data; and a learned evaluation model execution unit that uses the learned mathematical model obtained by the learning device and the set acquired by the interface unit to evaluate the motion shown in the motion video data included in the set.

6. A learning method comprising a control step of learning a learning object using motion video data, which is video data of an image showing the motion of an evaluation object which is a human or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion captured in the motion video data by a motion measurement sensor which is a sensor attached to at least one of the body of the evaluation object, an item worn by the evaluation object, and an item used by the evaluation object when performing the motion, and which obtains results according to the motion of the evaluation object, and ground truth data which indicates the result of evaluation of the motion, wherein the learning object is a mathematical model which evaluates the motion based on the motion video data and the motion measurement data.

7. An evaluation method comprising: an interface step of acquiring a pair of motion video data showing the motion of an evaluation subject and motion measurement data obtained during the motion shown in the motion video data; and a control step of learning a learning subject using motion video data, which is video data of a video showing the motion of an evaluation subject which is a human or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor which is a sensor attached to at least one of the body of the evaluation subject, an item worn by the evaluation subject, and an item used by the evaluation subject when performing the motion and which obtains a result according to the motion of the evaluation subject, and ground truth data which indicates the result of evaluation of the motion, wherein the learning subject is a mathematical model which evaluates the motion based on the motion video data and the motion measurement data, and a learned evaluation model execution step of evaluating the motion shown in the motion video data included in the set using the learned mathematical model obtained by the learning method and the set obtained by the interface step.

8. A program for causing a computer to function as either or both of the learning device according to any one of claims 1 to 4 and the evaluation device according to claim 5.

Citation Information

Patent Citations

  • Multi-modal dance action evaluation method

    CN110711374A

  • Method and system for evaluating the quality of a surgical procedure from in-vivo video

    US20170132785A1

  • Temporal fusion of multimodal data from multiple data acquisition systems to automatically recognize and classify an action

    US20170220854A1

  • Movement evaluating method, computer program, and movement evaluating system

    WO2022049700A1