Information provision method, information provision program, and scoring support system

Abstract

To improve the quality of providing information on a recognition result of skill.SOLUTION: In an information providing method, a computer executes processing for calculating feature quantity corresponding to a skill output by a machine learning model for skill recognition from skeleton information input into the machine learning model for skill recognition, calculating a difference between the calculated feature quantity and feature quantity obtained from training data to which a label of a skill same as the skill among training data of skeleton information used for training of the machine learning model for skill recognition, and outputting the calculated difference between feature quantities.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to information provision technology. [Background technology]

[0002] Human posture and behavior recognition are used in various fields. For example, in gymnastics, the current judging method is visually judged by multiple judges, but with the evolution of equipment and improvements in training methods, the movements are becoming more complex and the techniques are becoming more advanced, making it more difficult for judges to recognize the techniques. As a result, there are concerns about maintaining the fairness and accuracy of judging, such as the judges giving different scores to athletes.

[0003] Against this background, a scoring support system has been proposed that uses a 3D laser sensor to acquire 3D point cloud data of players, detects the players' skeletal information, recognizes the techniques performed by the players, and provides information on the results of the technique recognition to assist the referee in scoring.

[0004] For example, in a scoring support system, the likelihood output by the machine learning model used to recognize a technique is provided as part of the information provided about the technique recognition results. As one aspect of this, it alerts users to mistakes in automatic scoring in situations where the machine learning model is likely to output an incorrect technique label, such as in cases of over-detection or false positives. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-228100 A [Patent Document 2] Special Publication No. 2014-509011 [Patent Document 3] JP 2019-166311 A [Patent Document 4] US Patent Application Publication No. 2012 / 0214594 Summary of the Invention [Problem to be solved by the invention]

[0006] However, as will be described below, the above likelihood does not necessarily provide sufficient quality of information regarding the technique recognition results.

[0007] That is, the likelihood mentioned above merely corresponds to the difference between the skeletal information used as training data when training the machine learning model and the skeletal information input in a scoring situation, and does not match the true likelihood. In particular, the more types of classes the machine learning model classifies, and further, the greater the influence of individual differences on the classification accuracy of the machine learning model, the more difficult it becomes for the likelihood mentioned above to match the true likelihood. For example, even if the likelihood output by the machine learning model is low, the label may be correct, and even if the likelihood output by the machine learning model is high, the label may be incorrect. In addition, the judgment process by the machine learning model may be different from that of humans and may be a black box, so even if the likelihood output by the machine learning model is provided, it may not be suitable for explanation to humans.

[0008] In one aspect, an object of the present invention is to provide an information providing method, an information providing program, and a scoring support system that can improve the quality of information provided regarding technique recognition results. [Means for solving the problem]

[0009] In one aspect of the information provision method, a computer executes a process to calculate features corresponding to a technique output by a machine learning model for technique recognition from skeletal information input to the machine learning model for technique recognition, calculate a difference between the calculated features and features obtained from training data of the skeletal information used to train the machine learning model for technique recognition that is labeled with the same technique as the technique, and output the calculated difference in features. Effect of the Invention

[0010] According to one embodiment, it is possible to improve the quality of information provided regarding technique recognition results. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a scoring support system. [Diagram 2] FIG. 2 is a schematic diagram showing the skeletal recognition function. [Diagram 3] FIG. 3 is a schematic diagram showing the technique recognition function. [Figure 4] FIG. 4 is a diagram showing an example of a multi-angle view. [Diagram 5] FIG. 5 is a diagram showing an example of the automatic marking view. [Figure 6] FIG. 6 is a block diagram of an example of a functional configuration of the server device according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of the marking support view. [Figure 8] FIG. 8 is a diagram showing an example of the training support view. [Figure 9] FIG. 9 is a flowchart of the information providing process according to the first embodiment. [Figure 10] FIG. 10 is a block diagram of an example of a functional configuration of a server device according to the second embodiment. [Figure 11] FIG. 11 is a flowchart of an information providing process according to the second embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the information providing method, the information providing program, and the scoring support system according to the present application will be described with reference to the attached drawings. Each embodiment merely shows one example or aspect, and the numerical values, range of functions, and usage scenes are not limited by such examples. Each embodiment can be appropriately combined within a range that does not cause contradiction in the processing contents. EXAMPLES

[0013] <System configuration> Fig. 1 is a diagram showing an example of the configuration of a scoring support system. The scoring support system 1 shown in Fig. 1 acquires three-dimensional data of a subject, a performer 3, and supports the scoring of the techniques performed by the performer 3 through bone structure recognition, technique recognition, etc.

[0014] An example of such a technique is a performance in gymnastics, but other techniques may also include performances in figure skating, as well as movements in various sports, such as a baseball swing or a soccer shot.

[0015] As shown in FIG. 1, the scoring support system 1 may include a 3D laser sensor 5, a server device 10, and a client terminal 30.

[0016] The 3D (Three-Dimensional) laser sensor 5 is an example of a sensor device that uses an infrared laser or the like to measure the distance to an object, that is, the depth, for each pixel corresponding to a scanning point. For example, the 3D laser sensor 5 may be a depth imaging camera or a laser sensor using LIDAR (Light Detection and Ranging) technology, such as a MEMS (Micro-Electro-Mechanical Systems) mirror type laser sensor.

[0017] The server device 10 is an example of a computer that provides various services. As just one example, the server device 10 provides a scoring support service in which a skeleton recognition function 7, a technique recognition function 8, and a scoring support function 9 are packaged as shown in FIG. 1. For example, the skeleton recognition function 7 provides a function of detecting skeleton information such as the skeleton parts of the performer 3, for example, the positions of the joints, using a depth image measured by a 3D laser sensor 5. The technique recognition function 8 provides a function of recognizing a technique performed by the performer 3, using time-series data of the skeleton information obtained by the skeleton recognition by the skeleton recognition function 7. The scoring support function 9 provides a function of supporting scoring based on the technique recognition result by the technique recognition function 8.

[0018] The scoring support service can be provided by causing any computer to execute package software that realizes the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9. For example, server device 10 can provide skeleton recognition function 7, technique recognition function 8, and scoring support function 9 as a cloud service by implementing it as a SaaS (Software as a Service) type application. Additionally, server device 10 can also be implemented as a server that provides skeleton recognition function 7, technique recognition function 8, and scoring support function 9 on-premise.

[0019] The client terminal 30 is an example of a computer that receives the above-mentioned service. Such a client terminal 30 is used by people involved in gymnastics, such as judges and trainers of the performers 3, as merely an example of users of the above-mentioned service. For example, the client terminal 30 may be a desktop computer such as a personal computer. In addition, the client terminal 30 is not limited to a general-purpose information processing device such as a laptop computer, a mobile terminal device, or a wearable terminal, and may be an HMI (Human Machine Interface) customized for judging support.

[0020] 1 shows an example in which the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9 are provided as a package, but each function may be provided as a module. In this case, the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9 may be realized by different computers, and may be provided by the same vendor or different vendors.

[0021] <Bone structure recognition function> Fig. 2 is a schematic diagram showing the skeleton recognition function 7. As shown in Fig. 2, the skeleton recognition function 7 can be realized by a hybrid method that combines skeleton recognition using a machine learning model and fitting, just as an example.

[0022] For example, for skeletal recognition, a machine learning model7m that takes a depth image as input and outputs an estimate of 3D skeletal coordinates, such as a neural network based on a CNN (Convolutional Neural Network), can be used. The "3D skeletal coordinates" referred to here refer to the coordinates in three-dimensional space where joints such as the head, shoulders, spine, elbows, wrists, hips, knees, and ankles are located, and can correspond to an example of "skeletal information."

[0023] For training such a machine learning model 7m, a data set 7TR including training data associated with a depth image and a 3D skeletal coordinate of a correct answer label can be used. For example, the training data can be prepared by generating a depth image from the 3D skeletal coordinate of a gymnastics event using computer graphics or the like. Under such a data set, in the learning phase, the depth image is set as an explanatory variable of the machine learning model 7m, the label is set as the objective variable of the machine learning model 7m, and the machine learning model 7m can be trained according to any machine learning algorithm, for example, deep learning. For example, the parameters of the machine learning model 7m are updated based on the loss between the estimated value of the 3D skeletal coordinate output by the machine learning model 7m to which the depth image is input and the 3D skeletal coordinate of the correct answer label. This results in a trained machine learning model 7M. In the inference phase, multi-viewpoint depth images output from multi-viewpoint 3D laser sensors 5A to 5N installed to overcome occlusion by the gymnastics equipment and the performer 3 himself are input to the machine learning model 7M. The machine learning model 7M to which the multi-viewpoint depth images are input in this way outputs the 3D skeletal coordinate of the performer 3.

[0024] In fitting, the output of the 3D skeletal coordinates of the machine learning model 7M and the fitting results of the previous frame are used as initial values, and the human body model is fitted to the 3D point cloud that integrates multi-view depth images. For example, an evaluation function (likelihood) that indicates the degree of match between the coordinates of the 3D point cloud and the surface coordinates of the human body model is defined, and the 3D skeletal coordinates are determined by optimizing the joint angle that maximizes the likelihood.

[0025] The 3D laser sensor 5 and skeletal recognition function 7 realize 3D sensing that performs three-dimensional measurement of the movements of the actor 3 without using markers.

[0026] <Technique recognition function> Fig. 3 is a schematic diagram showing the technique recognition function 8. As shown in Fig. 3, the technique recognition function 8 can be realized by a machine learning model 8m, for example, a CNN-based neural network, that receives time-series data of 3D skeletal coordinates as input and outputs the name of a technique in gymnastics.

[0027] For training such a machine learning model 8m, a data set 8TR including training data associated with time series data of 3D skeletal coordinates and the names of techniques with correct labels can be used. For example, the training data can be time series data of 3D skeletal coordinates in which a performance is performed in a gymnastics competition. Under such a data set, in the learning phase, the 3D skeletal coordinates can be used as explanatory variables of the machine learning model 8m, the labels can be used as objective variables of the machine learning model 8m, and the machine learning model 8m can be trained according to any machine learning algorithm, such as deep learning. For example, the parameters of the machine learning model 8m are updated based on the loss between the estimated value of the technique name output by the machine learning model 8m to which the time series data of the 3D skeletal coordinates has been input and the technique name of the correct label. This results in a trained machine learning model 8M. In the inference phase, the time series data of the 3D skeletal coordinates obtained by skeletal recognition by the skeletal recognition function 7 is input to the machine learning model 8M. The machine learning model 8M to which the time series data of the 3D skeletal coordinates has been input in this way outputs the name of the technique performed by the performer 3 as the recognition result of the technique. At this time, the information regarding the technique recognition results may include information that can be identified by matching the technique name with rules and scoring rules, such as the technique group name, the difficulty level of the technique, and the difficulty evaluation score.

[0028] <Scoring support function> The scoring support function 9 can provide information to those involved in gymnastics based on the 3D skeleton coordinates obtained by the skeleton recognition function 7, the skill recognition results obtained by the skill recognition function 8, or a combination of these. For example, the scoring support function 9 can provide the multi-angle view shown in Fig. 4 and the automatic scoring view shown in Fig. 5 as examples of user interfaces for judges corresponding to examples of those involved in gymnastics. By implementing such a scoring support function 9 using a Web application, it is possible to provide the multi-angle view shown in Fig. 4 and the automatic scoring view shown in Fig. 5 via a browser running on the client terminal 30.

[0029] FIG. 4 is a diagram showing an example of the multi-angle view. As shown in FIG. 4, the multi-angle view 400 displays a skeleton model, in which the 3D skeleton coordinates of the actor 3 in each frame in which the 3D laser sensor 5 acquires a depth image, from multiple viewpoints such as the front, side, plane, and free viewpoint. Whether or not to display the skeleton model from multiple viewpoints in this way can be switched by operating the tabs 410 and 420. For example, when the “Multi” tab 410 is selected, the skeleton model of the actor 3 is displayed for each of multiple viewpoints as shown in FIG. 4. On the other hand, when the “Single” tab 420 is selected, the skeleton model of the actor 3 corresponding to the specified viewpoint is displayed. Furthermore, in the multi-angle view 400, the frame in which the skeleton model of the actor 3 is displayed can be specified by sliding the slider 430A on the seek bar 430. According to such a multi-angle view 400, the joint angles of the actor 3, which may be one of the scoring criteria, can be confirmed in detail.

[0030] FIG. 5 is a diagram showing an example of an automatic scoring view. As shown in FIG. 5, the automatic scoring view 500 includes a skeleton model view 510 and an automatic scoring table 520. For example, the skeleton model view 510 displays a skeleton model in which the 3D skeleton coordinates of the performer 3 at a user-specified angle are modeled. The frame in which the skeleton model of the performer 3 is displayed in the skeleton model view 510 can be specified by sliding a slider 530A on the seek bar 530. Furthermore, the automatic scoring table 520 displays the recognition results of the techniques in chronological order. In the example shown in FIG. 5, the technique name, technique group number, technique difficulty, and difficulty value are displayed in chronological order in the vertical direction of the window of the automatic scoring view 500, that is, from top to bottom. In addition to the information of each technique being displayed individually in this way, the scoring results of the D score for all performances are also displayed. Here, FIG. 5 shows an example in which the D score tab 540 is selected, but when the E score tab 550 is selected, it is possible to switch to displaying the scoring results of the E score for all performances. Such an automatic scoring view 500 can reduce the burden of umpire work.

[0031] In addition to the multi-angle view and the automatic scoring view, the likelihood of a technique output by the machine learning model 8M used for technique recognition may be provided as part of the provision of information on the technique recognition result. This has the aspect of drawing attention to errors in automatic scoring in situations where the machine learning model 8M is likely to output an incorrect technique label, such as in overdetection or false detection.

[0032] In addition, in Figures 4 and 5, contents for referees are given as an example of providing information on the results of technique recognition, but the target of information provision is not limited to referees. For example, contents for trainers or entertainment purposes may be provided.

[0033] <One aspect of the issue> As will be explained below, the likelihood output by the above machine learning model 8M is not necessarily of sufficient quality in providing information regarding the technique recognition results.

[0034] That is, the above likelihood merely corresponds to the difference between the skeletal information used as training data when training the machine learning model 8M and the skeletal information input in a scoring situation, and does not match the true likelihood. In particular, the more types of classes the machine learning model 8M classifies, and further, the greater the influence of individual differences on the classification accuracy of the machine learning model 8M, the more difficult it becomes for the above likelihood to match the true likelihood. For example, even if the likelihood output by the machine learning model 8M is low, the label may be correct, and even if the likelihood output by the machine learning model 8M is high, the label may be incorrect. In addition, the judgment process by the machine learning model 8M may be different from that of humans and may be a black box, so even if the likelihood output by the machine learning model 8M is provided, it may not be suitable for explanation to people.

[0035] <Configuration of Server Device 10> In this embodiment, therefore, a functional configuration of a server device 10 having an information providing function capable of improving the quality of information provision regarding technique recognition results will be described. Fig. 6 is a block diagram showing an example of a functional configuration of the server device 10 according to the first embodiment. In Fig. 6, blocks corresponding to functions possessed by the server device 10 are diagrammatically illustrated.

[0036] As shown in Fig. 6, the server device 10 has a communication interface unit 11, a storage unit 13, and a control unit 15. Although Fig. 6 illustrates functional units related to the scoring support service including the above-mentioned information provision function, this does not prevent modules corresponding to some functions from operating on different computers or virtual machines. In addition, the server device 10 may be provided with functions that existing computers are equipped with as defaults or options, not limited to the functional units illustrated in Fig. 6.

[0037] The communication interface unit 11 corresponds to an example of a communication control unit that controls communication with other devices, such as the 3D laser sensor 5 and the client terminal 30. As just one example, the communication interface unit 11 can be realized by a network interface card such as a LAN (Local Area Network) card. As one aspect, the communication interface unit 11 receives depth images from the 3D laser sensor 5 on a frame-by-frame basis, and outputs information related to technique recognition results to the client terminal 30.

[0038] The storage unit 13 is a functional unit that stores various data. As an example, the storage unit 13 is realized by a storage, for example, an internal, external, or auxiliary storage. For example, the storage unit 13 stores a data set 8TR of training data used to generate the machine learning model 8M. In addition to this data set 8TR, the storage unit 13 can store various data such as a data set 7TR of training data used to generate the machine learning model 7M, 3D skeletal coordinates, skill recognition results, or automatic scoring results.

[0039] The control unit 15 is a processing unit that performs overall control of the server device 10. For example, the control unit 15 is realized by a hardware processor. As shown in Fig. 6, the control unit 15 has an acquisition unit 15A, a skeleton recognition unit 15B, a technique recognition unit 15C, a feature amount calculation unit 15D, a feature amount difference calculation unit 15E, and a scoring support unit 15F.

[0040] The acquisition unit 15A is a processing unit that acquires a depth image. As an example, the acquisition unit 15A can acquire multi-viewpoint depth images output from the multi-viewpoint 3D laser sensors 5A to 5N in units of frames. Here, the information source from which the acquisition unit 15A acquires the depth image may be any information source, and is not limited to communication via the network NW. For example, the acquisition unit 15A may acquire the depth image from a storage included in the server device 10, or a removable medium that is detachable from the server device 10, such as a memory card or a USB (Universal Serial Bus) memory.

[0041] The skeleton recognition unit 15B is a processing unit that executes skeleton recognition. The skeleton recognition unit 15B may correspond to the skeleton recognition function 7 shown in FIG. 1. As an embodiment, the skeleton recognition unit 15B can use the trained machine learning model 7M shown in FIG. 2 for skeleton recognition of the performer 3. For example, when a new frame of a depth image is acquired by the acquisition unit 15A, the skeleton recognition unit 15B inputs a multi-viewpoint depth image corresponding to the new frame to the machine learning model 7M. Then, the skeleton recognition unit 15B outputs an estimated value of the 3D skeleton coordinates of the performer 3 output from the machine learning model 7M to which the multi-viewpoint depth image is input, to the technique recognition unit 15C.

[0042] The technique recognition unit 15C is a processing unit that executes technique recognition. The technique recognition unit 15C may correspond to the technique recognition function 8 shown in FIG. 1. In one embodiment, the technique recognition unit 15C can use the trained machine learning model 8M shown in FIG. 3 for technique recognition of the performer 3. For example, when the skeleton recognition unit 15B outputs 3D skeleton coordinates corresponding to a new frame, the technique recognition unit 15C inputs time series data of the 3D skeleton coordinates for a specific number of frames going back from the new frame to the machine learning model 8M. Then, the technique recognition unit 15C outputs the technique recognition result of the performer 3 output from the machine learning model 8M to which the time series data of the 3D skeleton coordinates has been input to the feature calculation unit 15D.

[0043] The feature amount calculation unit 15D is a processing unit that calculates feature amounts related to a technique. In one embodiment, the feature amount calculation unit 15D can start processing when a technique is recognized by the technique recognition unit 15C, for example, when the label output by the machine learning model 8M is not undetected, i.e., not "no technique". For example, the feature amount calculation unit 15D calculates feature amounts corresponding to each scoring item of the technique recognized by the technique recognition unit 15C for each scoring item. That is, since the type and number of scoring items defined in the scoring rules may differ depending on the type of technique, different types and numbers of feature amounts are calculated for each technique. For example, in the case of "front and back split jumps", feature amounts such as front and back splits, front knee angle, and jump height are calculated. Also, in the case of "front and back split ring jumps", feature amounts such as the height of the rear leg and the bending angle of the upper body are further calculated in addition to the feature amounts of the front and back splits, front knee angle, and jump height. For example, for each feature corresponding to a scoring item of a technique, the feature calculation unit 15D identifies a frame to be used for calculating the feature from the time series data of 3D skeletal coordinates input to the machine learning model 8M, and calculates the feature from the 3D skeletal coordinates in that frame.

[0044] The feature amount difference calculation unit 15E is a processing unit that calculates the difference in feature amount between the input data used for the technique recognition and the training data corresponding to the technique recognition result. For example, the feature amount calculated by the feature amount calculation unit 15D is used as the feature amount of the input data. On the other hand, the feature amount calculated from the time series data of the 3D skeleton coordinates corresponding to the training data to which the same technique label as the technique recognized by the technique recognition unit 15C is assigned, among the data set 8TR used for generating the machine learning model 8M, is used as the feature amount of the training data. Note that the feature amount of the training data can be calculated by the feature amount calculation unit 15D on demand, but a feature amount calculated in advance can be used as a preset.

[0045] More specifically, the feature difference calculation unit 15E calculates the feature difference for each of the L training data to which the same label of the technique as that recognized by the technique recognition unit 15C is assigned, and for each of the K types of feature corresponding to the number of scoring items of the technique recognized by the technique recognition unit 15C. The closer the value of the feature difference calculated in this way is to zero, the smaller the distance of the feature corresponding to the scoring item between the input data and the training data, or the higher the similarity of the feature. In other words, the smaller the value of the feature difference, the higher the possibility that the technique is performed according to the scoring item corresponding to the technique of the label assigned to the training data. Therefore, the likelihood that the technique is performed according to the scoring rules in gymnastics can be quantified. As a result, it is possible to provide an index that expresses the true certainty that the technique is performed and is compatible with the human judgment process.

[0046] Furthermore, the feature amount difference calculation unit 15E calculates the likelihood of the feature amount from the above-mentioned feature amount difference. Hereinafter, the likelihood of the feature amount may be referred to as "feature amount likelihood". As an example only, the feature amount likelihood can be calculated by normalizing the feature amount difference to a specific numerical range, for example, the range of 0 to 1. In this case, the reciprocal of the feature amount difference can be taken from the aspect of normalizing the feature amount likelihood such that the closer to 1 the feature amount is, the more likely the feature amount is, and the closer to 0 the feature amount is, the less likely the feature amount is.

[0047] After the K types of feature amount likelihoods are calculated in this manner, the feature amount difference calculation unit 15E can calculate one representative value, for example a statistical value, representing the K types of feature amount likelihoods as the likelihood of a technique. Hereinafter, the likelihood of a technique may be referred to as "technique likelihood". As merely one example, the minimum value of the K types of feature amount likelihoods may be derived as the technique likelihood. In addition, an average value such as an arithmetic mean or weighted mean of the K types of feature amount likelihoods, or a median value of the K types of feature amount likelihoods may be derived as the technique likelihood. By calculating such a technique likelihood for each of the L training data, the feature amount difference calculation unit 15E calculates the L technique likelihoods.

[0048] The scoring support unit 15F is a processing unit that supports scoring. The scoring support unit 15F may correspond to the scoring support function 9 shown in FIG. 1. As one aspect, the scoring support unit 15F can display the multi-angle view 400 shown in FIG. 4 or the automatic scoring view 500 shown in FIG. 5 on the client terminal 30. As another aspect, the scoring support unit 15F can provide the feature likelihood and the skill likelihood calculated by the feature difference calculation unit 15E to the client terminal 30 as part of the provision of information on the skill recognition result. For example, the scoring support unit 15F extracts the minimum skill likelihood from among the L skill likelihoods calculated by the feature difference calculation unit 15E, and extracts K types of feature likelihoods calculated from the training data corresponding to the minimum skill likelihood. Then, the scoring support unit 15F causes the client terminal 30 to display the minimum skill likelihood or the K types of feature likelihoods calculated from the training data corresponding to the minimum skill likelihood.

[0049] Fig. 7 is a diagram showing an example of the scoring support view. As shown in Fig. 7, the scoring support view 700 includes a skeleton model view 710 that displays a skeleton model in which the 3D skeleton coordinates of the performer 3 are modeled. The frame in which the skeleton model of the performer 3 is displayed in this skeleton model view 710 can be linked according to the position of a slider 720A on a seek bar 720. In the example of Fig. 7, the skeleton model of the skill "front and back split jump with front legs bent and straightened" being performed in the frame corresponding to the position of the slider 720A on the seek bar 720 is displayed in the skeleton model view 710.

[0050] Furthermore, the display of the symbol bar 730, which displays the performances performed by the performer 3 in chronological order, is linked to the position of the slider 720A on the seek bar 720. For example, the technique being performed in the frame corresponding to the position of the slider 720A on the seek bar 720 and the techniques performed before and after it are displayed in chronological order. In the example of Fig. 7, the symbols of the techniques "rondat", "backward somersault, both feet on the platform", "one turn standing on one leg", "front and back split jump with front legs bent and straightened" and "sissonne" are displayed in a horizontal direction, for example, from left to right.

[0051] Furthermore, the symbols included in the symbol bar 730 are displayed with their corresponding skill likelihoods. In the example of Fig. 7, the skill likelihood of "0.91" is displayed below the symbol "Rondato", the skill likelihood of "0.99" is displayed below the symbol "Backward somersault, both feet on the platform", and the skill likelihood of "0.87" is displayed below the symbol "One turn standing on one leg". Furthermore, the skill likelihood of "0.25" is displayed below the symbol "Front and back split jump with front legs bent and straightened", and the skill likelihood of "0.85" is displayed below the symbol "Sisonne".

[0052] Here, among the symbols and skill likelihoods displayed in the symbol bar 730, a distinction is made between the display form of a symbol or skill likelihood that satisfies a specific condition and the display form of other symbols or skill likelihoods. For example, a symbol or skill likelihood that satisfies either of the conditions of a skill likelihood being equal to or below a threshold value or a skill likelihood that is the lowest in the symbol bar 730 can be highlighted. In the example of FIG. 7, the symbol "front and back split jump with front legs bent and straightened" is displayed in hatching, and the font of the skill likelihood is displayed in bold.

[0053] Such a symbol bar 730 can increase the attention of "split jump with bent and straightened front legs" which is more likely to be over-detected or erroneously detected than other techniques among the technique recognition results. As a result, it is possible to draw attention to an automatic scoring error of "split jump with bent and straightened front legs". This makes it possible to take measures such as estimating and confirming whether "split jump with bent and straightened front legs" has been erroneously detected as "split jump" which has a similar movement to "split jump with bent and straightened front legs".

[0054] Fig. 8 is a diagram showing an example of a training support view. As shown in Fig. 8, a training support view 800 further includes a feature likelihood list 810 in addition to GUI components similar to the skeleton model view 710, the seek bar 720, the slider 720A, and the symbol bar 730 shown in Fig. 7.

[0055] In such feature amount likelihood list 810, feature amount likelihoods relating to feature amounts of a type corresponding to the score item of the technique being performed in the frame corresponding to the position of slider 720A on seek bar 720 are displayed in a list.

[0056] In the example of Figure 8, from the perspective that the technique being performed is a "front and back split jump with front leg bent and straightened," feature likelihood list 810 displays three feature likelihoods 1 to 3: front and back split "0.88," front knee angle "0.25," and jump height "0.98."

[0057] Furthermore, of the three feature amount likelihoods 1 to 3, the front knee angle "0.25" is highlighted because it is relatively lower than the other feature amount likelihoods, or satisfies the condition that it is lower than a threshold that can be defined in the numerical range of 0 to 1, for example, 0.4.

[0058] According to such feature likelihood list 810, it is possible to present to the performer 3 and his / her trainer, together with the skill recognition result "front and rear split jump with bent and straightened front leg", the fact that the front knee angle is "0.25", which is a relatively and absolutely low value. This makes it possible to suggest an area for improvement, that is, that the bending and straightening of the front leg is insufficient as stipulated in the scoring rules for "front and rear split jump with bent and straightened front leg".

[0059] <Processing flow> Fig. 9 is a flowchart showing the procedure of the information provision process according to the first embodiment. The process shown in Fig. 9 is merely an example and can be started when a new frame of a depth image is acquired. As shown in Fig. 9, the skeleton recognition unit 15B inputs the depth image acquired by the acquisition unit 15A to the machine learning model 7M, and causes the machine learning model 7M to output an estimated value of the 3D skeleton coordinates of the actor 3, thereby performing skeleton recognition (step S101).

[0060] Next, the technique recognition unit 15C performs technique recognition by inputting time series data of 3D skeletal coordinates for a specific number of frames going back from the new frame into the machine learning model 8M, and outputting the technique recognition result of the actor 3 to the machine learning model 8M (step S102).

[0061] At this time, if a technique is recognized in the technique recognition in step S102, for example, if the label output by the machine learning model 8M is not "undetected," that is, "no technique" (step S103 Yes), the feature calculation unit 15D executes the following process: That is, the feature calculation unit 15D calculates K types of feature corresponding to the scoring items of the technique recognized in the technique recognition in step S102 (step S104).

[0062] Then, the feature amount difference calculation unit 15E starts a loop process 1 in which the processes from step S105 to step S107 are repeated a number of times corresponding to the number L of pieces of training data to which the same label as the technique recognized in the technique recognition in step S102 has been assigned. Note that, although an example in which a loop process is performed is given here, the processes from step S105 to step S107 may be performed in parallel for each number L of training data.

[0063] Furthermore, the feature amount difference calculation unit 15E starts a loop process 2 in which the processes of steps S105 and S106 are repeated a number of times corresponding to the number K of feature amount types, which corresponds to the number of score items of the technique recognized in the technique recognition in step S102. Note that, here as well, the processes of steps S105 and S106 may be performed in parallel for each number K of feature amount types.

[0064] That is, the feature amount difference calculation unit 15E calculates the feature amount difference between the input data used for the technique recognition and the training data corresponding to the technique recognition result (step S105). For example, the feature amount calculated in step S104 is used as the feature amount of the input data. On the other hand, the feature amount calculated from the time series data of the 3D skeleton coordinates corresponding to the training data to which the same technique label as the technique recognized in the technique recognition in step S102 is assigned, among the data set 8TR used for generating the machine learning model 8M, is used as the feature amount of the training data.

[0065] Next, the feature amount difference calculation unit 15E calculates a feature amount likelihood from the feature amount difference calculated in step S105 (step S106).

[0066] By repeating such loop process 2, a feature likelihood can be obtained for each of the number K of types of feature. When loop process 2 ends, feature difference calculation unit 15E calculates one representative value representing the K types of feature likelihood, for example the minimum value, as a technique likelihood (step S107). Furthermore, by repeating loop process 1, a technique likelihood can be obtained for each number L of training data.

[0067] When loop processing 1 is completed, the scoring support unit 15F displays on the client terminal 30 the smallest skill likelihood among the L skill likelihoods obtained in loop processing 1 and K types of feature likelihoods calculated from the training data corresponding to the smallest skill likelihood (step S108), and then terminates the processing.

[0068] <One aspect of the effect> As described above, the server device 10 according to the present embodiment provides the difference between the feature amount calculated from the skeletal information input to the machine learning model for technique recognition and the feature amount calculated from the skeletal information of the training data for the same technique as the technique output by the machine learning model for technique recognition. Therefore, it is possible to provide information that expresses the true probability that the technique is being performed and also provides an index that is compatible with the human judgment process. Therefore, the server device 10 according to the present embodiment can improve the quality of information provided regarding the technique recognition results. EXAMPLES

[0069] In the above Example 1, an example was given in which feature differences, and therefore feature likelihood and technique likelihood, were calculated for each L number of training data items that have been labeled with the same technique as the technique recognized in technique recognition. However, it is not necessary to calculate feature differences, etc. for each L number of training data items.

[0070] In other words, when calculating the features, since the scoring items differ depending on the technique recognized by the technique recognition, processing is performed such as identifying the type of feature to be calculated and, for each type of feature, specifying the frames to be used to calculate the features from the time series data of the 3D skeletal coordinates.

[0071] In order to reduce such processing load, this embodiment describes an application example in which the difference in 3D skeletal coordinates is calculated between the input data used for technique recognition and the training data corresponding to the technique recognition result, and the training data with the smallest difference is narrowed down as the target for calculating the feature difference.

[0072] Fig. 10 is a diagram showing an example of a functional configuration of a server device 20 according to Example 2. In Fig. 10, functional units having the same functions as those of the server device 10 shown in Fig. 6 are denoted by the same reference numerals, and the description thereof will be omitted. As shown in Fig. 10, the server device 20 is different from the server device 10 shown in Fig. 6 in that the control unit 25 further includes a skeleton difference calculation unit 25A.

[0073] The skeleton difference calculation unit 25A is a processing unit that calculates the difference in 3D skeleton coordinates. Hereinafter, the difference in 3D skeleton coordinates may be referred to as the "skeleton difference."

[0074] As just one example, the skeleton difference calculation unit 25A can start processing when a technique is recognized by the technique recognition unit 15C, for example, when the label output by the machine learning model 8M is not undetected, that is, not no technique.

[0075] For example, the skeleton difference calculation unit 25A calculates the difference in 3D skeleton coordinates between the input data used for the technique recognition and the training data corresponding to the technique recognition result. For example, as the input data, the time series data of the 3D skeleton coordinates obtained by the skeleton recognition by the skeleton recognition unit 15B, or from another perspective, the time series data of the 3D skeleton coordinates used for the technique recognition by the technique recognition unit 15C, is used. On the other hand, as the training data, the time series data of the 3D skeleton coordinates corresponding to the training data to which the same technique label as the technique recognized by the technique recognition unit 15C is assigned, among the data set 8TR used for generating the machine learning model 8M, is used.

[0076] More specifically, the skeleton difference calculation unit 25A executes the following process for each of the L training data to which the same technique label as the technique recognized by the technique recognition unit 15C is assigned. That is, when the number of frames of the time series data of the 3D skeleton coordinates differs between the input data and the training data, the skeleton difference calculation unit 25A unifies the number of frames of both. When unifying the number of frames between the input data and the training data in this way, the number of frames of the larger one can be matched to the number of frames of the smaller one, or the number of frames of the smaller one can be matched to the number of frames of the larger one. For example, when the number of frames of the smaller one is matched to the number of frames of the larger one, resampling is required to interpolate the missing frames. On the other hand, when the number of frames of the larger one is matched to the number of frames of the smaller one, it is only necessary to thin out the excess frames. From this aspect, the skeleton difference calculation unit 25A unifies the number of frames of both the input data and the training data by thinning out the 3D skeleton coordinates corresponding to the excess number of frames from the time series data of the 3D skeleton coordinates of the input data and the training data that has the larger number of frames.

[0077] After unifying the number of frames, the skeletal difference calculation unit 25A calculates the difference in 3D skeletal coordinates between the input data and the training data, that is, the skeletal difference, for each of the unified M frames and for each number N of joints defined in the skeletal model. The "difference" calculated here may be an absolute value or a squared value. When the skeletal difference is calculated for each of the N joints, the skeletal difference calculation unit 25A calculates the sum of the N skeletal differences. After that, when the sum of the N skeletal differences is calculated for each of the M frames, the skeletal difference calculation unit 25A calculates the average value from the sum of the N skeletal differences for the M frames. By calculating the average value of the N skeletal differences as a representative value in this way, even if the number of frames M unified between the training data and the input data differs for each of the L training data, it is possible to eliminate the need to adjust the number of frames between the L training data. Then, when the average value of the N skeletal differences is calculated for each of the L training data, the skeletal difference calculation unit 25A selects the training data with the smallest average value of the N skeletal differences among the L training data.

[0078] This makes it possible to narrow down the L training data to the training data that has the smallest distance from the input data or the highest similarity to the input data.

[0079] Fig. 11 is a flowchart showing the procedure of the information provision process according to the second embodiment. The process shown in Fig. 11 is merely an example and can be started when a new frame of a depth image is acquired. As shown in Fig. 11, the skeleton recognition unit 15B inputs the depth image acquired by the acquisition unit 15A to the machine learning model 7M, and causes the machine learning model 7M to output an estimated value of the 3D skeleton coordinates of the actor 3, thereby performing skeleton recognition (step S101).

[0080] Next, the technique recognition unit 15C performs technique recognition by inputting time series data of 3D skeletal coordinates for a specific number of frames going back from the new frame into the machine learning model 8M, and outputting the technique recognition result of the actor 3 to the machine learning model 8M (step S102).

[0081] At this time, if a technique is recognized in the technique recognition in step S102, for example, if the label output by the machine learning model 8M is not undetected, i.e., not no technique (step S103 Yes), the skeleton difference calculation unit 25A executes the following processing.

[0082] That is, the skeleton difference calculation unit 25A starts a loop process 11 in which the process from step S201 to step S204 is repeated a number of times corresponding to the number L of training data to which the same label as the technique recognized in the technique recognition in step S102 has been assigned. Note that, although an example in which the loop process is performed is given here, the process from step S201 to step S204 may be performed in parallel for each number L of training data.

[0083] The skeleton difference calculation unit 25A unifies the number of frames of both the input data and the training data by thinning out the 3D skeleton coordinates corresponding to the excess number of frames from the time series data of the 3D skeleton coordinates of the input data and the training data, whichever has the larger number of frames (step S201). Note that if the number of frames is the same between the input data and the training data, the process of step S201 can be skipped.

[0084] Thereafter, the skeleton difference calculation unit 25A starts a loop process 12 in which the processes of steps S202 and S203 are repeated a number of times corresponding to the number of frames M unified in step S201. Note that, although an example in which a loop process is performed is given here, the processes of steps S202 and S203 may be performed in parallel for each number of frames M.

[0085] Furthermore, the skeleton difference calculation unit 25A starts a loop process 13 in which the process of step S202 is repeated a number of times corresponding to the number N of joints defined in the skeleton model. Note that, although an example in which the loop process is performed is given here, the process of step S202 may be performed in parallel for each number N of joints.

[0086] That is, the skeleton difference calculation unit 25A calculates the difference in 3D skeleton coordinates between the input data and the training data, that is, the skeleton difference (step S202). By repeating such loop processing 13, the skeleton difference can be obtained for each of the N joints.

[0087] When the loop process 13 ends, the skeleton difference calculation unit 25A calculates the sum of the N skeleton differences (step S203). By repeating such a loop process 12, the sum of the N skeleton differences can be obtained for each of the M frames.

[0088] When the loop process 12 is completed, the skeleton difference calculation unit 25A calculates an average value from the sum of N skeleton differences for M frames (step S204). By repeating such a loop process 11, the average value of N skeleton differences can be obtained for each of L training data.

[0089] Then, when the loop process 11 ends, the skeleton difference calculation unit 25A selects the training data having the smallest average value of the N skeleton differences among the L training data (step S205).

[0090] Thereafter, a loop process 14 is started in which the process from step S206 to step S208 is repeated a number of times corresponding to the number K of types of feature amounts corresponding to the scoring items of the technique recognized in the technique recognition in step S102. Note that, although an example in which a loop process is performed is given here, the process from step S206 to step S208 may be performed in parallel for each number K of types of feature amounts.

[0091] That is, the feature amount calculation unit 15D calculates feature amounts corresponding to the score items of the technique recognized in the technique recognition in step S102 (step S206).

[0092] Then, the feature amount difference calculation unit 15E calculates the difference in feature amount between the input data used for the technique recognition and the training data selected in step S205 (step S207). For example, the feature amount calculated in step S206 is used as the feature amount of the input data. On the other hand, the feature amount calculated from the time series data of the 3D skeleton coordinates corresponding to the training data selected in step S205 among the data set 8TR used to generate the machine learning model 8M is used as the feature amount of the training data.

[0093] Next, the feature amount difference calculation unit 15E calculates a feature amount likelihood from the feature amount difference calculated in step S207 (step S208).

[0094] By repeating such loop process 14, a feature likelihood can be obtained for each of the feature types K. When loop process 14 ends, feature difference calculation unit 15E calculates one representative value representing the K types of feature likelihoods, for example the minimum value, as a skill likelihood (step S209).

[0095] Finally, the scoring support unit 15F causes the client terminal 30 to display the skill likelihood calculated in step S209 and the K types of feature amount likelihood obtained in the loop process 14 (step S210), and ends the process.

[0096] As described above, the server device 20 according to this embodiment, like the server device 10 according to the above-mentioned first embodiment, calculates and outputs the difference in feature amounts corresponding to the scoring items of the technique between the input data used for technique recognition and the training data corresponding to the technique recognition result. Therefore, it is possible to express the true probability that the technique is being performed and to provide an index that is compatible with the human judgment process. Therefore, the server device 20 according to this embodiment can also realize improvement in the quality of information provision regarding the technique recognition result.

[0097] Furthermore, the server device 20 according to the present embodiment calculates the difference in 3D skeletal coordinates between the input data used for technique recognition and the training data corresponding to the technique recognition result, and narrows down the training data with the smallest difference as the target for feature difference calculation. Therefore, the server device 20 according to the present embodiment can reduce the processing load, such as identifying the type of feature to be calculated and specifying the frames to be used for feature calculation from the time-series data of 3D skeletal coordinates for each type of feature. EXAMPLES

[0098] Although the embodiments of the disclosed device have been described above, the present invention may be embodied in various different forms other than the above-described embodiments. Therefore, other embodiments included in the present invention will be described below.

[0099] <Training data> In the above-mentioned first embodiment, an example was given in which a feature likelihood and a technique likelihood were calculated for each of L training data items to which a label of the same technique as that recognized in the technique recognition in step S102 shown in Fig. 9 was assigned. In the above-mentioned second embodiment, an example was given in which a skeleton difference was calculated for each of L training data items to which a label of the same technique as that recognized in the technique recognition in step S102 shown in Fig. 11 was assigned. The subject of these calculations does not necessarily have to be the L training data items.

[0100] As just one example, a single training data representative of the L training data can be generated by calculating a statistical value of the 3D skeletal coordinates among the L training data, such as an arithmetic mean or a weighted mean, and the generated training data can be used as the calculation target. As another example, the difference in the 3D skeletal coordinates can be calculated for each pair of training data that can be extracted from the L training data, and the pair of training data with the largest difference in the 3D skeletal coordinates among all pairs can be used as the calculation target.

[0101] <Body frame difference> In the above-mentioned second embodiment, an example was given of using a skeleton difference to narrow down L training data to one training data. However, the skeleton difference selected in step S205 shown in FIG. 11 can be output together with the skill likelihood and feature likelihood output in step S210.

[0102] By providing such information on the skeletal difference, it is possible to deal with situations in which only the skill likelihood and feature likelihood are insufficient, as described below. For example, take the skill of a "forward roll" in which a person rotates forward from a sitting position, turns around, and then turns further forward to return to a sitting position. The features of this skill are "starts with feet on the ground," "rotates 180 degrees," "turns into a handstand position with hands on the ground," "rotates 180 degrees," and "ends with feet on the ground." In contrast, a situation may occur in which, when a person fails a skill and falls off a balance beam, the person quickly grabs the beam to avoid falling headfirst, then spins around once before falling. In this case, depending on the situation, the person may fall from a position where the feet are on the ground, grab the beam near where they have rotated 180 degrees, rotate another 180 degrees, and land on their feet, which may result in a movement similar to a forward roll when comparing the skill likelihood and feature likelihood. Even in such a case, the above-mentioned skeletal difference is likely to show a difference between a failed skill and a forward roll, and therefore can be used to distinguish between a failed skill and a forward roll.

[0103] <Distribution and integration> Also, each component of each device shown in the figure does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. For example, the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, or the marking support unit 15F may be connected via a network as an external device of the server device 10. Also, the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the skeleton difference calculation unit 25A, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, or the marking support unit 15F may be connected via a network as an external device of the server device 20. Alternatively, the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the feature calculation unit 15D, the feature difference calculation unit 15E, or the scoring support unit 15F may be included in another device, which is connected to a network and works together to realize the functions of the server device 10. Alternatively, the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the skeleton difference calculation unit 25A, the feature calculation unit 15D, the feature difference calculation unit 15E, or the scoring support unit 15F may be included in another device, which is connected to a network and works together to realize the functions of the server device 20.

[0104] <Hardware configuration> Moreover, the various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, an example of a computer that executes an information provision program having the same functions as those in the first and second embodiments will be described below with reference to FIG.

[0105] Fig. 12 is a diagram showing an example of a hardware configuration. As shown in Fig. 12, the computer 100 has an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Furthermore, the computer 100 has a CPU 150, a ROM 160, a HDD 170, and a RAM 180. These units 110 to 180 are connected via a bus 140.

[0106] As shown in FIG. 12, the HDD 170 may store an information provision program 170a that performs the same functions as the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, and the marking support unit 15F shown in the above-mentioned first embodiment. The HDD 170 may also store an information provision program 170a that performs the same functions as the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the skeleton difference calculation unit 25A, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, and the marking support unit 15F shown in the above-mentioned second embodiment. This information provision program 170a may be integrated or separated, similar to each component shown in FIG. 6 or FIG. 10. That is, the HDD 170 does not necessarily have to store all the data shown in the above-mentioned first and second embodiments, and it is sufficient that the data used for processing is stored in the HDD 170.

[0107] Under such an environment, the CPU 150 reads out the information provision program 170a from the HDD 170 and loads it in the RAM 180. As a result, the information provision program 170a functions as an information provision process 180a as shown in FIG. 12. The information provision process 180a loads various data read out from the HDD 170 in an area of ​​the storage area of ​​the RAM 180 that is assigned to the information provision process 180a, and executes various processes using the loaded various data. For example, the processes shown in FIG. 9 and FIG. 11 may be included as examples of the processes executed by the information provision process 180a. Note that in the CPU 150, all of the processing units shown in the above-mentioned first embodiment do not necessarily need to operate, and it is sufficient that the processing units corresponding to the processes to be executed are virtually realized.

[0108] The information provision program 170a does not necessarily have to be stored in the HDD 170 or the ROM 160 from the beginning. For example, the information provision program 170a is stored in a "portable physical medium" such as a flexible disk, a so-called FD, a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card that is inserted into the computer 100. The computer 100 may then obtain and execute the information provision program 170a from such a portable physical medium. The information provision program 170a may also be stored in another computer or a server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, or the like. The information provision program 170a thus stored may be downloaded to the computer 100 and then executed.

[0109] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0110] (Appendix 1) Calculating feature values ​​corresponding to techniques output by a machine learning model for technique recognition from skeletal information input to the machine learning model for technique recognition; Calculating a difference between the calculated feature amount and a feature amount obtained from training data of skeletal information used in training the machine learning model for recognizing the technique, the training data being labeled with the same technique as the technique; Output the calculated feature difference. An information providing method, characterized in that the processing is executed by a computer.

[0111] (Appendix 2) The computer further executes a process of calculating a skeletal difference between the skeletal information input to the machine learning model for technique recognition and a plurality of training data to which the same technique label as the technique output by the machine learning model for technique recognition is assigned; the process of calculating the difference in the feature amount includes a process of calculating a difference between a feature amount obtained from one training data among the plurality of training data, the training data having the smallest skeleton difference, and a feature amount calculated in the process of calculating the feature amount. 2. The information providing method according to claim 1,

[0112] (Appendix 3) The process of calculating the skeleton difference includes a process of calculating the skeleton difference between the skeleton information input to the machine learning model for technique recognition and a statistical value of the skeleton information calculated from the plurality of training data. 3. The information providing method according to claim 2,

[0113] (Appendix 4) The process of calculating the skeletal difference includes a process of calculating a skeletal difference for each pair of training data extracted from the plurality of training data, and calculating the skeletal difference between the training data of the pair having the largest skeletal difference among each pair and skeletal information input to the machine learning model for technique recognition. 3. The information providing method according to claim 2,

[0114] (Additional Note 5) The output process includes a process of outputting a likelihood of a feature amount obtained by normalizing a difference between the feature amounts. 2. The information providing method according to claim 1,

[0115] (Additional Note 6) The process of calculating the feature amount includes a process of calculating the feature amount for each type of feature amount corresponding to a technique output by the machine learning model for technique recognition, the process of calculating the difference in the feature amounts includes a process of calculating the difference in the feature amounts for each type of the feature amounts, the output process includes a process of outputting a statistical value representing a likelihood of a plurality of feature quantities obtained by normalizing differences between the feature quantities for each type of the feature quantity. 2. The information providing method according to claim 1,

[0116] (Appendix 7) Calculating feature values ​​corresponding to techniques output by a machine learning model for technique recognition from skeletal information input to the machine learning model for technique recognition; Calculating a difference between the calculated feature amount and a feature amount obtained from training data of skeletal information used in training the machine learning model for recognizing the technique, the training data being labeled with the same technique as the technique; Output the calculated feature difference. An information providing program that causes a computer to execute a process.

[0117] (Appendix 8) The computer is further caused to execute a process of calculating a skeletal difference between the skeletal information input to the machine learning model for technique recognition and a plurality of training data to which the same technique label as the technique output by the machine learning model for technique recognition is assigned; the process of calculating the difference in the feature amount includes a process of calculating a difference between a feature amount obtained from one training data among the plurality of training data, the training data having the smallest skeleton difference, and a feature amount calculated in the process of calculating the feature amount. 8. The information providing program according to claim 7,

[0118] (Appendix 9) The process of calculating the skeleton difference includes a process of calculating the skeleton difference between the skeleton information input to the machine learning model for technique recognition and a statistical value of the skeleton information calculated from the plurality of training data. 9. The information providing program according to claim 8,

[0119] (Appendix 10) The process of calculating the skeletal difference includes a process of calculating a skeletal difference for each pair of training data extracted from the plurality of training data, and calculating the skeletal difference between the training data of the pair having the largest skeletal difference among each pair and skeletal information input to the machine learning model for technique recognition. 9. The information providing program according to claim 8,

[0120] (Supplementary Note 11) The output process includes a process of outputting a likelihood of a feature amount obtained by normalizing a difference between the feature amounts. 8. The information providing program according to claim 7,

[0121] (Appendix 12) The process of calculating the feature amount includes a process of calculating the feature amount for each type of feature amount corresponding to a technique output by the machine learning model for technique recognition, the process of calculating the difference in the feature amounts includes a process of calculating the difference in the feature amounts for each type of the feature amounts, the output process includes a process of outputting a statistical value representing a likelihood of a plurality of feature quantities obtained by normalizing differences between the feature quantities for each type of the feature quantity. 8. The information providing program according to claim 7,

[0122] (Appendix 13) A sensor device for acquiring a depth image; a server device having: a skeleton recognition unit that performs skeleton recognition on the depth image; a technique recognition unit that recognizes techniques by inputting skeleton information obtained by the skeleton recognition into a machine learning model for technique recognition; a feature calculation unit that calculates a feature corresponding to a technique output by the machine learning model for technique recognition from the skeleton information input to the machine learning model for technique recognition; a feature difference calculation unit that calculates a difference between the calculated feature and a feature obtained from training data of skeleton information used to train the machine learning model for technique recognition, the training data being labeled with the same technique as the technique; and an information output unit that outputs the calculated feature difference to a client terminal; A scoring support system comprising:

[0123] (Appendix 14) The server device further includes a skeleton difference calculation unit that calculates a skeleton difference between skeleton information input to the machine learning model for technique recognition and a plurality of training data to which the same technique label as the technique output by the machine learning model for technique recognition is assigned; The scoring support system described in Appendix 13, wherein the feature difference calculation unit calculates the difference between a feature obtained from one training data among the plurality of training data in which the skeletal difference is smallest and a feature calculated by the process of calculating the feature.

[0124] (Appendix 15) The scoring support system described in Appendix 14, characterized in that the skeletal difference calculation unit calculates the skeletal difference between skeletal information input into the machine learning model for technique recognition and a statistical value of skeletal information calculated from the multiple training data.

[0125] (Appendix 16) The scoring support system described in Appendix 14, characterized in that the skeletal difference calculation unit calculates a skeletal difference for each pair of training data extracted from the multiple training data, and calculates the skeletal difference between the training data of the pair having the largest skeletal difference among each pair and skeletal information input to the machine learning model for technique recognition.

[0126] (Supplementary Note 17) The scoring support system according to Supplementary Note 13, wherein the information output unit outputs a likelihood of a feature in which a difference between the feature is normalized.

[0127] (Additional Note 18) The feature calculation unit calculates the feature for each type of feature corresponding to a technique output by the machine learning model for technique recognition; the feature amount difference calculation unit calculates a difference between the feature amounts for each type of the feature amount; The scoring support system described in Appendix 13, characterized in that the information output unit outputs a statistical value representing the likelihood of a plurality of feature quantities in which the differences between the feature quantities are normalized for each type of the feature quantity. [Explanation of symbols]

[0128] 1. Scoring support system 3. Actors 5 3D laser sensor 10. Server equipment 11 Communication interface section 13 Storage section 15 Control section 15A Acquisition Department 15B Skeleton Recognition Unit 15C Technique Recognition Department 15D Feature Calculation Unit 15E Feature Difference Calculation Unit 15F Scoring Support Department

Claims

1. Calculating feature amounts corresponding to techniques output by a machine learning model for technique recognition from skeletal information input to the machine learning model for technique recognition; Calculating a difference between the calculated feature amount and a feature amount obtained from training data of skeletal information used in training the machine learning model for recognizing the technique, the training data being labeled with the same technique as the technique; Output the calculated feature difference. An information providing method, characterized in that the processing is executed by a computer.

2. The computer further executes a process of calculating a skeletal difference between the skeletal information input to the machine learning model for technique recognition and a plurality of training data to which the same technique label as the technique output by the machine learning model for technique recognition is assigned; the process of calculating the difference in the feature amount includes a process of calculating a difference between a feature amount obtained from one training data among the plurality of training data, the training data having the smallest skeleton difference, and a feature amount calculated in the process of calculating the feature amount.

2. The information providing method according to claim 1,

3. the process of calculating the skeleton difference includes a process of calculating the skeleton difference between skeleton information input to the machine learning model for technique recognition and a statistical value of skeleton information calculated from the plurality of training data; 3. The information providing method according to claim 2,

4. the process of calculating the skeletal difference includes a process of calculating a skeletal difference for each pair of training data extracted from the plurality of training data, and calculating the skeletal difference between the training data of the pair having the largest skeletal difference among each pair and skeletal information input to the machine learning model for technique recognition; 3. The information providing method according to claim 2,

5. the outputting process includes a process of outputting a likelihood of a feature amount obtained by normalizing a difference between the feature amounts.

5. The information providing method according to claim 1, wherein the information providing method comprises:

6. the process of calculating the feature amount includes a process of calculating the feature amount for each type of feature amount corresponding to a technique output by the machine learning model for technique recognition, the process of calculating the difference in the feature amounts includes a process of calculating the difference in the feature amounts for each type of the feature amounts, the output process includes a process of outputting a statistical value representing a likelihood of a plurality of feature quantities obtained by normalizing differences between the feature quantities for each type of the feature quantity.

5. The information providing method according to claim 1, wherein the information providing method comprises:

7. Calculating feature amounts corresponding to techniques output by a machine learning model for technique recognition from skeletal information input to the machine learning model for technique recognition; Calculating a difference between the calculated feature amount and a feature amount obtained from training data of skeletal information used in training the machine learning model for recognizing the technique, the training data being labeled with the same technique as the technique; Output the calculated feature difference. An information providing program that causes a computer to execute a process.

8. A sensor device for acquiring a depth image; a server device having: a skeleton recognition unit that performs skeleton recognition on the depth image; a technique recognition unit that recognizes techniques by inputting skeleton information obtained by the skeleton recognition into a machine learning model for technique recognition; a feature calculation unit that calculates a feature corresponding to a technique output by the machine learning model for technique recognition from the skeleton information input to the machine learning model for technique recognition; a feature difference calculation unit that calculates a difference between the calculated feature and a feature obtained from training data of skeleton information used to train the machine learning model for technique recognition, the training data being labeled with the same technique as the technique; and an information output unit that outputs the calculated feature difference to a client terminal; A scoring support system comprising:

Images