Skeleton estimation device, skeleton estimation method, and gymnastics scoring assistance system

Abstract

The skeleton estimation device (100) acquires, in units of frames, estimation information that estimates a skeleton change in a prescribed period, calculates a feature quantity (a) of the skeleton change in the prescribed period, calculates an approximation line (b) based on the feature quantity (a), calculates a correlation state of the calculated feature quantity (a) and the approximation line (b) using a correlation coefficient, and detects, as a bad skeleton, a frame portion (T1, T2) in which an estimation anomaly is estimated in the estimation information of the prescribed period based on the correlation coefficient. For example, the skeleton estimation device (100) detects, as a bad skeleton, a part of frames in the estimation information of the skeleton estimated by a skill of one or more turns in a prescribed period from the start (ts) to the end (te) of the skill of the gymnast. By re-estimating the frame portion of the bad skeleton, the evaluator can be prompted with estimation information of the skeleton that is correctly estimated.

Description

Technical Field

[0001] This invention relates to a skeleton estimation device, a skeleton estimation method, and a gymnastics scoring assistance system for estimating changes in the human skeleton. Background Technology

[0002] Human skeleton recognition is used in a wide range of fields, including gymnastics and medicine. For example, current gymnastics scoring methods rely on visual evaluation by multiple judges. However, due to the increasing sophistication of techniques in recent years, it has become increasingly difficult to score solely through visual evaluation. Therefore, a skeleton estimation device is being considered to identify the athlete's skeletal information, joint angles, and executed techniques based on 3D data obtained from a 3D laser sensor. This provides judges with skeletal (e.g., posture) information, thereby assisting in scoring.

[0003] Previously, techniques related to skeleton estimation included generating contour images from frames of a time series, evaluating the probability of each pose state using the generated contour images based on pose state prediction, and estimating the pose of a multi-jointed animal. Other techniques included predicting joint positions in order corresponding to the connectivity constraints between joints in a kinematic model, projecting the predicted positions onto a 2D image, and identifying the pose or movement of a human subject by evaluating the reliability of the projected positions, for motion capture. Additionally, techniques included generating and correcting multi-joint movements in computer graphics by using motion patterns of human joints stored in a database during human animation generation (see, for example, Patent Documents 1-3 below).

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2009-015558

[0007] Patent Document 2: Japanese Patent Application Publication No. 2007-333690

[0008] Patent Document 3: Japanese Patent Application Publication No. 10-171854 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] However, in existing technologies, movements of the human skeleton, such as detecting and correcting abrupt changes in the skeleton (joints) in adjacent frames, cannot detect changes in the skeleton based on a series of (overall) movements of the human body.

[0011] As a specific example, in gymnastics rotation techniques, the body rotates once within the overall technique. In this case, existing technology only detects abrupt changes in skeleton (posture) between adjacent frames, failing to detect changes in the skeleton of the body rotating once within the overall technique. In gymnastics, there are also cases with varying rotation amounts and instances where there is no rotation. Therefore, for example, in existing technology, changes in skeleton between adjacent frames are possible movements of the human body, but if a skeleton that differs from the actual movement is estimated, it cannot be detected as an undesirable skeleton. Thus, in existing technology, undesirable skeletons that should be corrected within the device cannot be detected in a series of movements performed within a specified period, such as gymnastics techniques.

[0012] On the one hand, the purpose of this invention is to accurately detect changes in the skeletal structure during a series of human movements.

[0013] Methods for solving problems

[0014] According to one aspect of the present invention, the component includes a control unit that acquires estimated information on skeleton changes within a specified period on a frame-by-frame basis, calculates a feature quantity of the skeleton changes within the specified period, calculates an approximation line based on the calculated feature quantity, calculates a correlation state between the calculated feature quantity and the approximation line, and detects the frame portion of the estimated information within the specified period that is estimated to be abnormal as a bad skeleton based on the correlation state.

[0015] Invention Effects

[0016] According to one aspect of the present invention, it is possible to accurately detect changes in the skeletal structure during a series of human movements. Attached Figure Description

[0017] Figure 1 This is an explanatory diagram of a skeleton determination example of a skeleton estimation device according to an embodiment.

[0018] Figure 2 This is a block diagram illustrating a functional example of a skeleton estimation device.

[0019] Figure 3 This is a diagram illustrating an example of the hardware configuration of a skeleton estimation device.

[0020] Figure 4 This is a flowchart illustrating a processing example of a skeleton estimation device.

[0021] Figure 5 This is a diagram showing an example of detecting a defective skeleton based on skeleton information estimated within a specified period.

[0022] Figure 6This is a diagram illustrating a structural example of a gymnastics scoring assistance system including a skeleton estimation device according to an embodiment.

[0023] Figure 7 This is a block diagram illustrating a functional example of defective skeleton detection for each feature quantity.

[0024] Figure 8 This is an illustration of case 1, where the feature quantity is less than the range.

[0025] Figure 9 This is an illustration of case 3, where the characteristic quantity exceeds its range. (Case 1)

[0026] Figure 10 This is an illustration of case 3, where the characteristic quantity exceeds its range. (Part 2)

[0027] Figure 11A This is an illustration of case 2, where the characteristic quantity is within the range. (Case 1)

[0028] Figure 11B This is an illustration of case 2, where the characteristic quantity is within the range. (Case 2)

[0029] Figure 12 This is a flowchart illustrating a processing example for detecting defective skeletons for each feature quantity.

[0030] Figure 13 This is an illustrative diagram for detecting defective skeletons based on other feature quantities. Detailed Implementation

[0031] (Implementation Method)

[0032] The implementation of the disclosed skeletal assessment device, skeletal estimation method, and gymnastics scoring assistance system will now be described in detail with reference to the accompanying drawings.

[0033] Figure 1 This is an explanatory diagram illustrating a skeleton determination example of the skeleton estimation device according to the embodiment. The skeleton estimation device 100 of the embodiment is capable of accurately detecting changes in the human skeleton during movement or the like over a specified period. For example, the skeleton estimation device 100 estimates the skeleton of an athlete that changes in accordance with the skills performed by the athlete during gymnastics competition. Here, the skeleton estimation device 100 of the embodiment detects portions of the skeleton estimated during movement (skills) that differ from the actual movements of the human body (athlete) as defective skeletons.

[0034] Information related to changes in the skeleton over a specified period is input to the skeleton estimation device 100. This information includes, for example, images (3D point groups) output frame-by-frame by the 3D laser sensors over the specified period, containing distance information for each of the orthogonal X, Y, and Z axes. The skeleton estimation device 100 acquires the outputs of multiple 3D laser sensors that capture images of the human body (athlete) from different locations.

[0035] The control unit 110 of the skeleton estimation device 100 estimates the changes in the human skeleton due to movement or other factors during a specified period by executing a skeleton estimation program. The control unit 110 estimates the movements (angles) of each joint of the human body based on information obtained related to changes in the skeleton caused by human movement or other factors during the specified period, and outputs skeleton information as the estimation result. The movement during the specified period is a pre-planned movement, such as gymnastics. In gymnastics, athletes perform skills within a specified period, and corresponding to these skills, the athlete's skeleton changes during the specified period.

[0036] Then, the skeleton estimation device 100 of the embodiment detects the parts of the skeleton information of the movement of each joint of the human body estimated through movement over a predetermined period that are different from the actual movement of the human body as defective skeletons. Thus, for example, the skeleton estimation device 100 can re-determine the skeleton of the parts of the estimation results that are different from the actual movement of the human body.

[0037] The control unit 110 focuses on the characteristic quantities (motion characteristic quantities) of human movements within a specified period. Based on the correlation with the expected motion characteristic quantities, it estimates the probability of the estimated skeleton, thereby detecting defective skeletons. Expected motion characteristic quantities are, for example, the characteristic quantities corresponding to skills performed in gymnastics competitions over a specified period. Therefore, the estimating results of the skeleton corresponding to the performance of the gymnast's skill can be provided to the judges. Furthermore, defective skeletons detected within the skeleton estimation device 100 can be re-evaluated. Thus, the estimating results of the skeleton corresponding to the athlete's skill can be provided to the judges of the competition with high accuracy.

[0038] Using gymnastics as an example, the processing outline of the control unit 110 is explained. (1) The 3D skeleton information of the athlete is obtained from a 3D laser sensor or the like. (2) The feature values ​​of each frame are calculated up to a specified period (e.g., from the start of the skill ts to the end te).

[0039] (3) Calculate the likelihood of the motion characteristics of the skill as a whole over the specified period. For example, the calculation method of likelihood can be divided into three cases, namely Case 1 to Case 3, according to the magnitude of the motion characteristics of the skill as a whole.

[0040] Case 1: For example, if the motion feature quantity is less than a preset range, the likelihood is calculated based on the maximum and minimum values ​​of the feature quantity in each frame.

[0041] Scenario 2: When the motion feature is within a pre-defined range, calculate an approximate line based on a multidimensional function across multiple pre-defined segmentation intervals. Then, calculate the likelihood based on the maximum correlation coefficient between the approximate line calculated for each segmentation interval and the motion feature.

[0042] Scenario 3: When the motion feature quantity exceeds the preset range, calculate the approximate line based on the feature quantity from the start ts to the end te of the technique.

[0043] In cases 2 and 3, the likelihood is calculated based on the correlation coefficient between the calculated approximate line and the motion characteristic quantity.

[0044] (4) When the likelihood is less than a preset threshold, calculate the frame with the largest difference between the approximate line and the motion feature quantity. Thus, the frame with the largest error (the frame with the largest error) can be detected in the estimated skeleton information of the bad skeleton part.

[0045] (5) In addition, it is possible to perform skeleton determination again on frames with bad skeletons based on the frame with the maximum error.

[0046] As characteristic quantities of the aforementioned movements, there are "rotational volume" and "tumbling volume" in gymnastics. Additionally, as multidimensional functions, there are linear functions (straight lines) and cubic functions (cubic curves). A cubic curve can be used for "rotational volume," and a straight line can be used for "tumbling volume."

[0047] Figure 1 (a) and (b) show the relationship between the approximate line and the motion characteristic quantity in scenario 3 above. The horizontal axis is time, and the vertical axis is the cumulative rotation angle. Here, the motion characteristic quantity is the "motion rotation quantity," which represents the cumulative rotation angle in a skillful second aerial position, such as a vault, where the body rotates more than once.

[0048] The control unit 110 calculates an approximate cubic curve b (dashed line in the figure) suitable for detecting rotation based on the cumulative rotation angle a (solid line in the figure) during a specified period, i.e., from the start ts to the end te of the technique. Next, the control unit 110 calculates the correlation coefficient between the cumulative rotation angle and the approximate cubic curve.

[0049] Next, the control unit 110 calculates the likelihood based on the correlation coefficient using an exponential function with preset parameters. Then, the control unit 110 detects the presence or absence of defective frames based on the similarity (correlation coefficient) between the cumulative rotation angle 'a' and the approximately cubic curve 'b'. If the frame is correct, such as... Figure 1As shown in (a), the cumulative rotation angle 'a' is similar to the approximately cubic curve 'b', with a high correlation coefficient. Therefore, the control unit 110 determines that the skeleton information estimated during the specified period (the period from the start 'ts' to the end 'te' of the technique) is correct.

[0050] On the other hand, in cases of skeletal errors, such as Figure 1 As shown in (b), the cumulative rotation angle 'a' differs from the approximate cubic curve 'b', exhibiting a low correlation coefficient. Therefore, the control unit 110 determines that the estimated skeleton information within the specified period (the period from the start 'ts' to the end 'te' of the technique) contains errors. In this case, the control unit 110 can, for example, perform a second skeleton determination for the periods T1 and T2 where the cumulative rotation angle 'a' differs from the approximate cubic curve 'b' and the correlation coefficient is low.

[0051] Thus, according to the implementation method, focusing on the human body's movements within a specified period, such as the motion characteristics of movements as a whole in gymnastics, the probability of an estimated skeleton is estimated based on its correlation with the expected motion characteristics. Therefore, it is possible to detect whether there are any defective skeletons among the estimated skeletons within the specified period.

[0052] Furthermore, the ability to re-evaluate the skeleton of detected defective parts can improve the accuracy of the estimated skeleton throughout the competition. For example, it can improve the accuracy of the skeleton estimation results corresponding to the athlete's skills, thus providing feedback to the competition's judges.

[0053] Figure 2 This is a block diagram illustrating a functional example of a skeleton estimation device. Figure 2 In the above context, "movement rotation amount" refers to the function that becomes the main function in the implementation, i.e., the motion characteristic quantity shown in case 3 above is "motion rotation amount", which is a function that involves turning more than once as a skill, such as the function corresponding to the skeleton estimation of the cumulative rotation angle in the second aerial position of the vault.

[0054] The skeleton estimation device 100 includes a skeleton acquisition unit 201, a feature calculation unit 202, an approximate cubic curve calculation unit 203, a likelihood calculation unit 204, and a defective skeleton detection unit 205. The skeleton acquisition unit 201 acquires 3D skeleton information of the athlete from a 3D laser sensor or the like. The 3D skeleton information is, for example, a depth image containing distance information along the X, Y, and Z axes over a defined period, such as multiple frames between the start (ts) and end (te) of a skill.

[0055] The skeleton estimation device 100 performs learned skeleton recognition on the acquired 3D skeleton information, calculates 3D joint coordinates, and fits the human body to these 3D joint coordinates. Based on the fitted joint coordinates, it identifies changes in the human body's movements (skills).

[0056] The feature quantity calculation unit 202, the approximate cubic curve calculation unit 203, the likelihood calculation unit 204, and the defective skeleton detection unit 205 are components related to skill recognition. The feature quantity calculation unit 202 calculates the feature quantities of each frame from the start ts to the end te of the skill. The approximate cubic curve calculation unit 203, corresponding to case 3, addresses the situation where the motion feature quantities exceed a predetermined range, and calculates an approximate line based on a multidimensional function (see reference 205) based on the feature quantities from the start ts to the end te of the skill. Figure 1 ).

[0057] The likelihood calculation unit 204 calculates the likelihood based on the correlation coefficient between the calculated approximation line and the motion feature quantity. The defective skeleton detection unit 205 calculates the frame with the largest difference between the approximation line and the motion feature quantity when the likelihood is lower than a preset threshold. Thus, frames containing defective skeleton portions in the estimated skeleton information are detected.

[0058] Figure 3 This is a diagram illustrating an example of the hardware structure of a skeleton estimation device. The skeleton estimation device 100 described above can use... Figure 3 The computer device shown is composed of hardware.

[0059] For example, the skeleton estimation device 100 includes a CPU (Central Processing Unit) 301, a memory 302, a network interface (IF) 303, a recording medium IF 304, and a recording medium 305. A bus 300 connects the various components.

[0060] CPU 301 is an arithmetic processing unit that functions as a control unit responsible for the overall processing of the skeleton estimation device 100. Memory 302 includes non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory) storing the program of CPU 301. Volatile memory is, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) used as the working area of ​​CPU 301.

[0061] Network IF303 is an interface for communication with networks 310 such as LAN (Local Area Network), WAN (Wide Area Network), and the Internet. The skeleton estimation device 100 can communicate with 3D laser sensors, scorers' terminals, etc., via this network IF303.

[0062] Recording medium IF304 is an interface used for reading and writing information processed by CPU 301 between recording medium 305 and recording medium 305. Recording medium 305 is a recording device for auxiliary storage 302. Recording medium 305 can be, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a USB (Universal Serial Bus) flash drive.

[0063] By executing the program recorded in the memory 302 or the recording medium 305 by the CPU 301, it is possible to achieve Figure 2 The functions of the skeleton estimation device 100 shown (control unit 110).

[0064] Figure 3 The hardware structure shown can also be applied to the judge's terminal (e.g., PC, smartphone, etc.). For example, the gymnastics scoring assistance system formed by connecting the skeleton estimation device 100 of the embodiment to the judge's terminal can be implemented using a general hardware structure.

[0065] Figure 4 This is a flowchart illustrating a processing example of a skeleton estimation device. Figure 4 This illustrates the operation performed by the control unit 110 of the skeleton estimation device 100 in conjunction with... Figure 2 The corresponding processing example is configured.

[0066] First, the control unit 110 obtains the 3D skeleton information of the contestant from a 3D laser sensor, etc. (step S401). Next, the control unit 110 calculates the feature quantities of each frame from the start ts to the end te of the technique (step S402). Next, based on the feature quantities from the start ts to the end te of the technique, the control unit 110 approximates a line based on a multidimensional function (approximately a cubic curve, referencing...). Figure 1 Perform the calculation (step S403).

[0067] Next, the control unit 110 calculates the likelihood based on the correlation coefficient between the calculated approximation line and the motion feature quantity (step S404). If the likelihood is less than a preset threshold, the control unit 110 calculates the frame with the largest difference between the approximation line and the motion feature quantity. Thus, frames containing portions of the defective skeleton in the estimated skeleton information are detected (step S405).

[0068] Therefore, it is possible to detect the presence or absence of poor skeletons estimated during the specified period from the start (ts) to the end (te) of a skill. Furthermore, it is possible to re-evaluate the skeletons of the detected poor skeletons, thereby improving the accuracy of the skeletons estimated throughout the entire specified period. For example, for a judge's terminal in a competition, this can improve and provide feedback on the accuracy of the skeleton estimation results corresponding to the competitor's skill.

[0069] Figure 5 This is a diagram showing an example of a defective skeleton detected based on skeleton information estimated over a specified period. Figure 5 (a) is a series of frames F1 to F11 from the prescribed period (the period from the start ts to the end te) of a gymnast performing a vaulting twist in gymnastics competition. Figure 5 (b) is related to Figure 5 (a) corresponds to each frame, representing CG data K1~K11 of the skeleton information of the contestant estimated by the skeleton estimation device 100.

[0070] The skeleton estimation device 100 uses the 3D skeleton information of the contestant obtained from the 3D laser sensor to perform learning-type skeleton recognition on the depth image containing distance information of the X, Y and Z axes, calculates 3D joint coordinates, and fits the 3D joint coordinates to the human body model. Figure 5 (b) The CG data K1 to K11 are the skeleton information estimated after fitting. The skeleton estimation device 100 detects the presence or absence of defective skeletons based on the estimated skeleton information (CG data K1 to K11).

[0071] exist Figure 5 In the actual image shown in (a), the vaulter uses a twisting technique to perform 1.5 rotations to the left and right within one somersault (vertical rotation). This is particularly evident in the rotation between F3 and F6. In contrast, in... Figure 5 In the skeleton information estimated in (b), CG data K3 to K6 are in a state without rotation.

[0072] Skeleton estimation device 100 through Figure 4 The processing shown above detects defective skeletons in CG data K3-K6 that differ from the actual movement. Therefore, regarding the estimated skeleton information, although the joint changes between adjacent frames are not unusual movements for the human body, it is possible to detect skeleton information (CG data K3-K6) that differs from the actual movement as defective skeletons (NG).

[0073] The skeleton estimation device 100 can correct the skeleton information of the defective skeleton NG portion to the correct skeleton information by performing a second skeleton estimation on the skeleton information (CG data K3 to K6) portion of the detected defective skeleton NG.

[0074] (An example of the application of a gymnastics scoring assistance system)

[0075] Figure 6This is a diagram illustrating a structural example of a gymnastics scoring assistance system including a skeleton estimation device according to an embodiment. The control unit 110 of the skeleton estimation device 100 provides skeletal information (e.g., posture) of the skills performed by the athlete during a specified period to the judges who score the skills performed by the athlete in gymnastics competitions, as information useful for scoring.

[0076] As shown in the figure, a 3D laser sensor 601 captures images of athlete H performing gymnastic skill A, and outputs multiple frames of an image (3D point group) containing distance information along each orthogonal X, Y, and Z axis to a skeleton estimation device 100 within a specified period (from the start ts to the end te of the skill). For example, multiple 3D laser sensors 601 are configured to capture images of athlete H, thereby outputting multi-viewpoint images to the skeleton estimation device 100.

[0077] The skeleton estimation device 100 includes a skeleton recognition unit 602, a machine learning unit 603, a fitting unit 604, and a skill recognition unit 605. The skeleton estimation device 100 can access a skill database (DB) 660.

[0078] The machine learning unit 603 uses the depth image 631 and joint coordinates 632 to maintain the 3D skeleton information as a learning model 633 and outputs it to the skeleton recognition unit 602.

[0079] The skeleton recognition unit 602 performs frame-by-frame recognition of the skeleton of contestant H during the performance of contestant H's skill, and outputs the 3D joint coordinates of contestant H frame by frame. The skeleton recognition unit 602 includes a depth image generation unit 621, a deep learning unit 622, and a 3D joint coordinate generation unit 623.

[0080] The depth image generation unit 621 generates multi-view depth images for contestant H based on information from 3D point groups output by multiple 3D laser sensors 601. The deep learning unit 622 accesses the learning model 633 of the machine learning unit 603 to learn the multi-view depth images generated by the depth image generation unit 621, obtaining 3D skeleton information. The 3D joint coordinate generation unit 623 generates 3D joint coordinate information for each frame of contestant H.

[0081] The fitting unit 604 performs a process for each frame that fits the 3D joint coordinates output by the skeleton recognition unit 602 (3D joint coordinate generation unit 623) to the depth image of the contestant H. At this time, the fitting unit 604 recognizes the 3D joint coordinates output by the 3D joint coordinate generation unit 623 as a primary skeleton 641.

[0082] The fitting unit 604 performs a position search on each joint of the primary skeleton 641 in the direction of the arrow in the figure to fit the depth image (torso) of the athlete H, thereby obtaining the fitted 3D joint coordinates 642. The fitted 3D joint coordinates 642 are more accurate skeletal information based on the primary skeleton and the body shape of the athlete H, and are used as the estimated skeletal information (posture) mentioned above.

[0083] The skill recognition unit 605 accesses the skill database 660 for various gymnastics skills and determines the sport (skill performed by athlete H) that matches the skill in the skill database 660. The skill recognition unit 605 maintains a skill recognition table 651 that pre-sets the characteristic quantities (sports characteristic quantities) and conditions for each skill name.

[0084] In the above-described embodiment, the detection of defective skeletons based on the estimated skeleton information is performed by the skill recognition unit 605. During defective skeleton detection, the skeleton estimation device 100 performs fitting processing based on the fitting unit 604 again on the frames of defective skeletons, such as the information of the CG data K3 to K6 portions described above.

[0085] The skill recognition unit 605 outputs information such as the athlete's skeletal structure, the angles of each joint, and the skill information performed as skeletal (e.g., posture) information to the judges.

[0086] (Structural examples for detecting defective skeletons for various features)

[0087] Figure 7 This is a block diagram illustrating a functional example of defective skeleton detection for each feature quantity. Figure 7 The text describes examples of defective skeletons, illustrating the magnitude (rotational volume) of each characteristic quantity. Figure 7 The defective skeleton detection function shown is set in Figure 6 The skill recognition unit 605 shown.

[0088] exist Figure 7 In the middle, to and Figure 2 The same structures are labeled with the same reference numerals. Figure 7 The skeleton estimation device 100 shown includes a skeleton estimation device 100 with Figure 2 The same skeleton acquisition unit 201, feature quantity calculation unit 202, approximate cubic curve calculation unit 203, likelihood calculation unit 204, and defective skeleton detection unit 205 are included. Additionally, a calculation method determination unit 703, a calculation interval storage unit 704, a multiple interval approximate cubic curve calculation unit 705, a multiple correlation coefficient calculation unit 706, a correlation coefficient calculation unit 707, a maximum and minimum value calculation unit 708, and an error calculation unit 709 are included.

[0089] Here, the calculation method determination unit 703 determines the likelihood calculation method that varies depending on the magnitude of the overall motion characteristic quantity of the skill. In the following examples, the cases are divided into three cases, namely Case 1 to Case 3, according to the magnitude of the motion characteristic quantity.

[0090] Case 1: For example, when the motion feature quantity is less than a preset range, the likelihood is calculated based on the maximum and minimum values ​​of the feature quantity in each frame. In Case 1, the maximum and minimum value calculation unit 708 is activated.

[0091] Case 2: When the motion feature quantity is within a preset range, an approximate line based on a multidimensional function is calculated for each of the preset multiple segmentation intervals. Then, the likelihood is calculated based on the largest correlation coefficient among the multiple calculated approximate lines and the motion feature quantity. In Case 2, the calculation interval storage unit 704, the multiple interval approximate cubic curve calculation unit 705, and the multiple correlation coefficient calculation unit 706 function.

[0092] Case 3: When the motion characteristic quantity exceeds a preset range, an approximate line based on a multidimensional function is calculated according to the characteristic quantities from the start (ts) to the end (te) of the technique. In Case 3, the approximate cubic curve calculation unit 203 and the correlation coefficient calculation unit 707 function. Case 3 corresponds to... Figure 2 Functional configuration.

[0093] The skeleton acquisition unit 201 acquires the contestant's 3D skeleton information from a 3D laser sensor, etc. The feature calculation unit 202 calculates the feature values ​​of each frame from the start ts to the end te of the technique.

[0094] The calculation method determination unit 703 distinguishes between case 1 to case 3 based on the magnitude of the characteristic quantity calculated by the characteristic quantity calculation unit 202, so that the functional units corresponding to each case 1 to case 3 can perform their functions.

[0095] If the magnitude of the feature quantity is less than a preset range, the calculation method determination unit 703 determines it to be Case 1, and the corresponding maximum / minimum value calculation unit 708 is activated. Furthermore, if the magnitude of the feature quantity is within the preset range, the calculation method determination unit 703 determines it to be Case 2, and the corresponding multiple interval approximate cubic curve calculation unit 705 and multiple correlation coefficient calculation unit 706 are activated. Additionally, if the magnitude of the feature quantity exceeds the preset range, the calculation method determination unit 703 determines it to be Case 3, and the corresponding approximate cubic curve calculation unit 203 and correlation coefficient calculation unit 707 are activated.

[0096] In Case 1, the characteristic quantity is equivalent to a range, such as in a vault where there is no turning movement. In this case, the maximum and minimum value calculation unit 708 calculates the maximum and minimum values ​​of the characteristic quantity between the start ts and the end te of the technique.

[0097] Case 2 corresponds to a situation where the characteristic value is within a certain range, such as a vaulting maneuver with a rotation of 1 / 2 times. In this case, the multiple interval approximate cubic curve calculation unit 705 calculates the approximate cubic curve in each of the segmented intervals stored in the calculation interval storage unit 704, that is, in each segmented interval obtained by dividing the predetermined period from the start ts to the end te of the maneuver into multiple segments. In addition, the multiple correlation coefficient calculation unit 706 calculates the correlation coefficient between the cumulative rotation angle and the approximate cubic curve for each segmented interval.

[0098] In case 3, the characteristic value exceeds the range, for example, in a vaulting maneuver with more than one rotation. In this case, the approximate cubic curve calculation unit 203 calculates an approximate line based on a multidimensional function according to the characteristic values ​​from the start (ts) to the end (te) of the maneuver. Additionally, the correlation coefficient calculation unit 707 calculates the correlation coefficient between the cumulative rotation angle and the approximate cubic curve.

[0099] The likelihood calculation unit 204 calculates the likelihood based on the outputs of cases 1 to 3, according to the correlation coefficient between the calculated approximation line and the motion feature quantity. The error calculation unit 709 calculates the difference (error) between the likelihood calculated by the likelihood calculation unit 204 and a preset threshold. The defective skeleton detection unit 205 calculates the frame with the largest difference between the approximation line and the motion feature quantity when the error likelihood is smaller than the preset threshold. Thus, the frame with the largest error in the estimated skeleton information can be detected. Furthermore, skeleton estimation can be performed again on the frame with the largest error.

[0100] (Example of handling scenario 1)

[0101] Figure 8 This is an illustration of case 1, where the feature quantity is less than the range. Figure 8 The horizontal axis represents time, and the vertical axis represents the cumulative rotation angle. Figure 8 In this context, it indicates a technique without rotation, such as the cumulative rotation angle in the second aerial phase of the vault. As a gymnastic technique definition, a final cumulative rotation angle of 90° or more is considered a 1 / 2 rotation technique, and one or more rotations are considered a technique with rotation. Therefore, a technique without rotation has a final cumulative rotation angle of less than 90°. The maximum and minimum value calculation section 708 calculates the maximum and minimum rotation angles during the specified period from the start (ts) to the end (te) of the technique.

[0102] Here, even if the final cumulative rotation angle is less than 90° and is therefore judged as having no rotation technique, for example, ... Figure 8 As shown, there is also a period (T1) during which the angle temporarily becomes 90° or more midway through the area where the skill was performed. In this case, although it is temporarily 90° or more, a reversal is performed in the air when the final accumulated rotation angle is less than 90°. This is an impossible movement for a human body. In this case, the likelihood calculation unit 204 outputs a low likelihood based on the accumulated amount of characteristic quantities during the specified period, thereby enabling the defective skeleton detection unit 205 to detect it as a defective skeleton.

[0103] (Example of handling scenario 3)

[0104] Figure 9 , Figure 10 This is an illustration of case 3 where the characteristic quantity exceeds the range. It describes situations involving more than one skillful turn, for example, the cumulative turn angle in the second aerial position of a vault. Figure 9 (a) and (b) are graphs showing examples of the correlation coefficients between the approximate curve and the motion characteristic quantity in Case 3. The horizontal axis represents time, and the vertical axis represents the cumulative rotation angle. The approximate cubic curve calculation unit 203 calculates the approximate cubic curve b (dashed line) based on the cumulative rotation angle a (solid line in the figure) between the start ts and end te of the technique over a specified period. The correlation coefficient calculation unit 707 calculates the correlation coefficient between the cumulative rotation angle a and the approximate cubic curve b.

[0105] The likelihood calculation unit 204 calculates the likelihood based on the correlation coefficient using an exponential function of preset parameters. Figure 10 This shows an example of an exponential function for calculating likelihood based on the correlation coefficient. Figure 10 The horizontal axis represents the correlation coefficient, and the vertical axis represents the likelihood. For example... Figure 10 As shown, an exponential function is used to perform the following transformation: the likelihood becomes a high value before the correlation coefficient reaches a preset value X, and the likelihood decreases sharply when it falls below the preset value Y.

[0106] Then, the defective skeleton detection unit 205 detects the presence or absence of a defective skeleton based on the similarity (correlation coefficient) between the cumulative rotation angle and the approximate cubic curve. If the skeleton is correct, such as... Figure 9 As shown in (a), the cumulative rotation angle 'a' is similar to the approximately cubic curve 'b', with a high correlation coefficient. Therefore, the control unit 110 determines that the skeleton information estimated during the specified period (the period from the start 'ts' to the end 'te' of the technique) is correct.

[0107] On the other hand, in cases of skeletal errors, such as Figure 9As shown in (b), the cumulative rotation angle 'a' differs from the approximate cubic curve 'b', exhibiting a low correlation coefficient. Therefore, the defective skeleton detection unit 205 determines that there is a portion of the skeleton information estimated during a specified period (the period from the start 'ts' to the end 'te' of the technique). In this case, the defective skeleton detection unit 205 can, for example, re-evaluate the skeleton during periods T1 and T2 where the cumulative rotation angle 'a' differs from the approximate cubic curve 'b' and the correlation coefficient is low.

[0108] (Example of handling scenario 2)

[0109] Figure 11A , Figure 11B This is an illustration of Case 2, where the characteristic quantity is within the range. It represents the case where the trick is a 1 / 2 turn, such as the cumulative turn angle in the second aerial position of a vault. In Case 2, the detection of the characteristic quantity is basically done using the same method as for more than one turn in Case 3 above. However, since the number of turns is small in 1 / 2 turn tricks, it is considered that there may be patterns where the turn is not performed in the entire trick but only near the beginning of the interval in which the trick is performed, or only near the end of the interval in which the trick is performed.

[0110] Figure 11A This is an illustration of a technique where a rotation is performed only near the very beginning of the overall technique. Figure 11B These are illustrations of techniques where a rotation is performed only near the very end of a complete technique. The horizontal axis of these diagrams represents time, and the vertical axis represents the cumulative rotation angle.

[0111] exist Figure 11A , Figure 11B In the pattern shown, when calculating the correlation coefficient between the cumulative rotation angle 'a' and the approximate cubic curve 'b' in the overall technique employed, such as these... Figure 11A (a) Figure 11B As shown in (a), the likelihood decreases.

[0112] Therefore, when the technique is a 1 / 2 turn, the focus is not on the entire interval where the technique was performed, but rather on dividing the period from the start (ts) to the end (te) of the technique into multiple segments, with a portion of each segment overlapping in time. For example, as... Figure 11A As shown, the curve is divided into three intervals, which serve as the first half of the 2 / 3 interval and the second half of the 2 / 3 interval. An approximate cubic curve is calculated in each interval.

[0113] For example, in a technique where the rotation is only performed near the beginning of the overall movement, such as... Figure 11AAs shown in (b), approximate cubic curves are calculated for the first two-thirds of the segmentation interval from the start of the technique (ts) to the end of period (t2), and for the second two-thirds of the segmentation interval from period (t1) to the end of period (te). This increases the likelihood in the first two-thirds of the segmentation interval, enabling high-precision detection of defective skeletons for estimating the skeletal information of the technique performed in the first half of the turn.

[0114] Similarly, in the case where a twisting technique is only performed near the very end of the overall technique, it is also like... Figure 11B As shown in (b), approximate cubic curves are calculated in the first two-thirds of the segmented interval and the second two-thirds of the segmented interval, respectively. As a result, the likelihood in the second two-thirds of the segmented interval is higher, enabling high-precision detection of defective skeletons based on the estimated skeleton information of the body rotation implemented in the second half.

[0115] Figure 12 This is a flowchart illustrating a processing example for detecting defective skeletons of various feature quantities. It shows the operation performed by the control unit 110 of the skeleton estimation device 100. Figure 7 Functions and Figures 8 to 11B The corresponding processing examples are provided for each of the explanations.

[0116] First, the control unit 110 obtains the 3D skeleton information of the contestant from a 3D laser sensor or the like (step S1201). Next, the control unit 110 calculates the feature values ​​of each frame from the start ts to the end te of the skill (step S1202).

[0117] Next, the control unit 110 determines the magnitude of the feature quantity and proceeds to the processing of large and small values ​​(step S1203). If the feature quantity is less than a preset range, it is determined to be Case 1 (step S1203: Case 1), and the process proceeds to step S1204. If the feature quantity is within the preset range, it is determined to be Case 2 (step S1203: Case 2), and the process proceeds to step S1205. If the feature quantity exceeds the preset range, it is determined to be Case 3 (step S1203: Case 3), and the process proceeds to step S1207.

[0118] In case 1, the control unit 110 calculates the maximum and minimum values ​​of the characteristic quantities of the specified period (from the start ts to the end te of the technique) (step S1204) and moves to the processing in step S1209.

[0119] In scenario 2, the control unit 110 calculates an approximate line based on a multidimensional function for a specified period (from the start of the technique to the end of the technique, ts) according to multiple pre-stored intervals (step S1205). Then, the control unit 110 calculates the correlation coefficient between the approximate line calculated in each of the multiple intervals and the motion characteristic quantity (step S1206), and proceeds to step S1209.

[0120] In case 3, the control unit 110 calculates an approximate cubic curve based on the characteristic quantity of the period up to the specified period (from the start ts to the end te of the technique) (step S1207). Then, the control unit 110 calculates the correlation coefficient between the calculated cubic approximate curve and the characteristic quantity (step S1208) and proceeds to the processing in step S1209.

[0121] Next, the control unit 110 calculates the likelihood (step S1209). For the processed data in cases 2 and 3, the likelihood is calculated based on the correlation coefficient between the calculated approximate line and the motion characteristic quantity. On the other hand, for the processed data in case 1, the control unit 110 does not use correlation but bases the likelihood on the characteristic quantity (maximum and minimum values ​​of the rotation angle). Figure 8 (Explanation), outputs a lower likelihood for impossible motions.

[0122] Then, if the likelihood is less than a preset threshold, the control unit 110 calculates the error between the approximate line and the motion feature quantity (step S1210). Then, the control unit 110 detects frames of defective skeletons in the estimated skeleton information by calculating frames with an error of more than a predetermined value (step S1211).

[0123] Therefore, the control unit 110 can detect the presence or absence of defective skeletons within a specified period (from the start ts to the end te of the technique). For example, Figure 5 CG frames K3 to K6 can be detected as defective skeletons (NG). Therefore, the control unit 110 can perform a second skeleton determination on the detected defective skeletons (NG).

[0124] (Example of detecting poor skeletons using other techniques)

[0125] In the above embodiments, taking the vault in gymnastics as an example, an example of detecting defective skeletons when the motion characteristic quantity is "motion rotation volume" is described. In these embodiments, the detection is not limited to gymnastics; it is possible to detect defective skeletons estimated based on changes in joint movement within the human body.

[0126] Figure 13 This is an illustrative diagram for detecting defective skeletons based on other feature quantities. In this... Figure 13The diagram shows the approximate line and the correlation state between the motion characteristic quantity and the motion characteristic quantity when the cumulative somersault angle motion characteristic quantity of the second aerial situation of the vault is "somersault (vertical rotation)".

[0127] When the motion characteristic is "somersault," a linear motion can be used to obtain the correlation between the cumulative somersault angle and the linear motion. In this case, the control unit 110 calculates an approximate linear motion (linear motion) based on the cumulative somersault angle. Then, the control unit 110 calculates the correlation coefficient between the cumulative somersault angle and the approximate linear motion. Afterward, the control unit 110 calculates the likelihood based on the correlation coefficient using an exponential function with pre-set parameters. Here, as an example of an exponential function for calculating the likelihood based on the correlation coefficient, it is possible to use an exponential function with... Figure 10 The same exponent coefficients are shown for the rotation cases.

[0128] Furthermore, the control unit 110 is capable of detecting defective skeletons based on the skeleton estimated based on the similarity (correlation) between the cumulative somersault angle and the approximate straight line. Figure 13 In the example shown in (a), the cumulative somersault angle 'a' is similar to the approximate straight line 'b', with a high correlation coefficient, thus indicating that the estimated skeleton is correct. On the other hand, as... Figure 13 (b) In this case, when there is a difference between the cumulative somersault angle a and the approximate straight line b, the correlation coefficient becomes lower, which can detect poor skeletons with estimated skew errors.

[0129] In this way, even with different features such as various techniques, it is possible to detect bad skeletons based on the correlation between the cumulative angle and the multidimensional function of the suitable technique during the specified period (the period from the start ts to the end te of the technique).

[0130] According to the implementation described above, the skeleton estimation device acquires estimated information about skeleton changes within a specified period on a frame-by-frame basis, calculates feature quantities of skeleton changes within the specified period, calculates an approximation line based on the calculated feature quantities, calculates the correlation state between the calculated feature quantities and the approximation line, and detects frames with estimation abnormalities in the estimated information within the specified period as defective skeletons based on the correlation state. The feature quantities correspond to changes in the skeleton, such as body rotation or turning. Therefore, it is possible to detect portions of the estimated skeleton that differ from the actual movement as defective skeletons. Furthermore, defective skeletons are determined based on the overall skeleton changes within the specified period. Thus, compared to conventional detection based on abrupt changes between adjacent frames, it is possible to detect defective skeletons that change over the entire specified period, even if changes are minimal between adjacent frames.

[0131] Furthermore, the skeleton estimation device detects defective skeletons using different methods based on the magnitude of the feature quantities. This allows for the detection of defective skeletons in the estimated information of different skeletons differentiated by rotational volume. When a feature quantity exceeds a preset range, an approximate curve based on a multidimensional function is calculated using the feature quantity over a specified period. A likelihood is calculated based on the correlation coefficient between the calculated feature quantity and the approximate curve. If the likelihood is lower than a preset threshold, the frame with the largest difference between the feature quantity and the approximate curve is identified as a defective skeleton. This enables accurate detection of skeleton defects in cases of large skeleton variations.

[0132] Furthermore, when the feature values ​​are within a preset range, the skeleton estimation device calculates an approximate curve based on a multidimensional function for each segmented interval obtained by dividing the specified period into multiple segments. It then calculates the likelihood based on the correlation coefficient between the feature values ​​calculated in each segmented interval and the approximate curve. If the likelihood is less than a preset threshold, the frame with the largest difference between the feature values ​​and the approximate curve is detected as a defective skeleton. Alternatively, the segmented intervals can overlap temporally. This allows for accurate detection of defective skeletons in each segmented interval. For example, even when the rotation only occurs in the first or second half of the specified period, defective skeletons for the estimated rotation information can be accurately detected by using segmented intervals.

[0133] Furthermore, when the feature quantity is less than a preset range, the skeleton estimation device calculates the maximum and minimum values ​​of the feature quantity over a specified period. It then calculates the likelihood based on the cumulative amount of the feature quantity, including the calculated maximum and minimum values, and uses this likelihood to detect defective skeletons. Therefore, even when the variation in the feature quantity is small, defective skeletons corresponding to the estimated information can be accurately detected. For example, it can accurately detect defective skeletons in the estimated information for a skeleton when no rotation occurs during the specified period.

[0134] Furthermore, the skeleton estimation device can re-estimate the skeleton of the frame portion where a faulty skeleton is detected. This allows for the provision of correct skeleton information to the outside throughout the specified period.

[0135] Furthermore, the aforementioned skeleton estimation device can be applied to gymnastics scoring assistance systems for gymnastics competitions where athletes perform techniques. The gymnastics scoring assistance system includes: a skeleton recognition unit that captures images of the athlete using a 3D laser sensor; the skeleton estimation device acquires images of 3D point groups containing distance information frame by frame; generates depth images based on the 3D point group images; and identifies the athlete's 3D skeleton based on a learning model pre-learned from the depth images; a fitting unit that outputs a 3D skeleton suitable for the athlete's skeleton as estimation information based on the 3D point groups and the 3D skeleton; and a control unit. The control unit pre-sets feature quantities for each technique and detects defective skeletons in the estimation information output by the fitting unit corresponding to the prescribed techniques performed by the athlete. Thus, it estimates the changes in the athlete's skeleton during a specified period and provides scorers with skeleton information corresponding to the techniques performed by the athlete during that period. Scorers are not limited to visual observation; they can also use the skeleton information provided by the skeleton estimation device to make more accurate scores based on the gymnastics techniques performed by the athlete, such as the angles of each joint.

[0136] In addition, the skeleton estimation device provides estimation information to the judges of gymnastics competitions. When detecting poor skeletons, it provides estimation information after re-estimating the skeleton of the frame portion with poor skeletons. Thus, the skeleton estimation device can always provide the judges with correctly estimated skeleton information, which can improve the reliability of the estimated skeleton information provided by the skeleton estimation device.

[0137] As described above, according to the embodiment, it is possible to accurately detect skeletons that change throughout the entire specified period, and even if a portion of the estimated skeleton information contains a defective skeleton, the defective skeleton can be detected and the skeleton re-estimated. The skeleton estimation of the embodiment is not limited to gymnastics; it can be applied to skeleton estimation in various fields such as patient examination and monitoring in medicine.

[0138] Furthermore, the skeleton estimation method described in the embodiments of the present invention can be implemented by executing a pre-prepared program on a processor such as a server. This skeleton estimation method is recorded on a computer-readable recording medium such as a hard disk, floppy disk, CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disk), or flash memory, and is executed by a computer reading it from the recording medium. Additionally, this skeleton estimation method can also be published via a network such as the Internet.

[0139] Symbol Explanation

[0140] 100 skeleton estimation device

[0141] 110 Control Department

[0142] 201 skeleton acquisition section

[0143] 202 Characteristic Quantity Calculation Department

[0144] 203 Approximate Cubic Curve Calculation Section

[0145] 204 Likelihood Calculation Department

[0146] 205 Defective Skeleton Detection Department

[0147] 301CPU

[0148] 302 memory

[0149] 303 Network Interface

[0150] 305 Recording Media

[0151] 310 Network

[0152] 601 3D Laser Sensor

[0153] 602 Skeleton Recognition Unit

[0154] 603 Machine Learning Department

[0155] 604 Fitting Section

[0156] 605 Skills Identification Department

[0157] 633 Learning Model

[0158] 651 Skill Identification Table

[0159] 660 Skills Database

[0160] 703 Calculation Method Judgment Department

[0161] 704 Computational Area Storage Unit

[0162] Calculation section for approximate cubic curves in more than 705 intervals

[0163] 706+ correlation coefficient calculation sections

[0164] 707 Correlation Coefficient Calculation Department

[0165] 708 Maximum and Minimum Value Calculation Section

[0166] 709 Error Calculation Department

[0167] Multiple frames from F1 to F11

[0168] K1~K11 CG Data

[0169] NG poor skeleton

Claims

1. A skeleton estimation device characterized by comprising: having a control section, The control section performs the following processing: estimation information that estimates a change in a skeleton in a prescribed period is acquired on a frame-by-frame basis, a feature quantity of a change in a skeleton in the prescribed period is calculated, an approximation line is calculated from the calculated feature quantity, a likelihood between the calculated feature quantity and the approximation line is calculated, based on the likelihood, a portion of the frames in which an abnormality is estimated in the estimation information of the prescribed period is detected as a bad skeleton.

2. The skeleton estimation device according to claim 1, wherein in a case where the feature quantity exceeds a range set in advance, the control section calculates an approximation curve based on a multi-dimensional function from the feature quantity in the prescribed period, the control section calculates a likelihood for the feature quantity in the prescribed period from a correlation coefficient between the calculated feature quantity and the approximation curve, in a case where the likelihood is less than a threshold value set in advance, the control section detects the frame in which a difference between the feature quantity and the approximation curve is the largest as the bad skeleton.

3. The skeleton estimation device according to claim 1, wherein in a case where the feature quantity is within a range set in advance, the control section calculates an approximation curve based on a multi-dimensional function from the feature quantity for each of divided intervals into which the prescribed period is divided, the control section calculates a likelihood from a correlation coefficient between the feature quantity calculated for each of the divided intervals and the approximation curve, in a case where the likelihood is less than a threshold value set in advance, the control section detects the frame in which a difference between the feature quantity and the approximation curve is the largest as the bad skeleton.

4. The skeleton estimation device according to claim 3, wherein the control section divides a plurality of the divided intervals in such a manner that a part of the divided intervals overlap each other in time.

5. The skeleton estimation device according to claim 1, wherein in a case where the feature quantity is less than a range set in advance, the control section calculates a maximum value and a minimum value of the feature quantity in the prescribed period, the control section calculates a likelihood based on an accumulated amount of the feature quantity including the calculated maximum value and minimum value, the control section detects the bad skeleton based on the likelihood.

6. The skeleton estimation device according to claim 1, wherein the control section performs estimation of a skeleton again for the portion of the frames of the detected bad skeleton.

7. The skeleton estimation device according to claim 1, wherein the feature quantity is a motion feature quantity corresponding to a prescribed motion of a human body assumed in advance in the prescribed period.

8. A skeleton estimation method characterized by comprising: by a computer performing the following processing: estimation information that estimates a change in a skeleton in a prescribed period is acquired on a frame-by-frame basis, a feature quantity of a change in a skeleton in the prescribed period is calculated, an approximation line is calculated from the calculated feature quantity, a likelihood between the calculated feature quantity and the approximation line is calculated, based on the likelihood, a portion of the frames in which an abnormality is estimated in the estimation information of the prescribed period is detected as a bad skeleton.

9. A gymnastics scoring support system for a gymnastics competition in which a performer performs a skill, the gymnastics scoring support system characterized by having a control section that performs the following processes: acquiring, in units of frames, estimation information that estimates a change in a skeleton of the performer over a prescribed period, calculating a feature quantity of a change in a skeleton over the prescribed period, calculating an approximation line based on the calculated feature quantity, calculating a likelihood between the calculated feature quantity and the approximation line, based on the likelihood, detecting a portion of the frames in which the estimation information of the prescribed period estimates an abnormality as a bad skeleton.

10. The gymnastics scoring support system according to claim 9, characterized in that: the gymnastics scoring support system has a 3D laser sensor that photographs the performer to output, in units of frames, an image of a 3-dimensional point cloud that contains distance information, and a skeleton estimation device, the skeleton estimation device has: a skeleton recognition section that generates a depth image from the image of the 3-dimensional point cloud, and recognizes a 3-dimensional skeleton of the performer based on a learning model that is learned in advance from the depth image; and a fitting section that outputs, as estimation information, a 3-dimensional skeleton that fits the skeleton of the performer based on the 3-dimensional point cloud and the 3-dimensional skeleton, the control section sets in advance a feature quantity for each of the skills, and detects the bad skeleton in the estimation information output by the fitting section in correspondence with performance of a prescribed skill of the performer.

11. The gymnastics scoring support system according to claim 10, characterized in that: the control section prompts a judge of the gymnastics competition with the estimation information, when the bad skeleton is detected, the control section prompts the estimation information after re-performing estimation of the skeleton for the portion of the frames of the bad skeleton.

12. The gymnastics scoring support system according to claim 11, characterized in that: the feature quantity is a movement feature quantity that differs depending on the skill performed by the performer.

13. The gymnastics scoring support system according to claim 12, characterized in that: the skill includes a turn and a flip. ​

Images