Determination method, determination program, and information processing device.

Abstract

This information processing device detects a plurality of segmentation points on the basis of a plurality of frames that each include a first feature quantity indicating a feature quantity regarding a subject's joint. The information processing device integrates two or more segmentation points among the plurality of segmentation points. The information processing device adjusts the segmentation points to be integrated so that a second feature quantity identified from the first feature quantities of the plurality of frames included in the section for the integrated segmentation points satisfies the conditions of a feature quantity of a prescribed basic motion.

Description

Technical Field

[0001] The present invention relates to a determination method and the like.

Background Art

[0002] In the field of gymnastics competitions, it is required to accurately evaluate the performance of athletes. Conventionally, multiple judges have visually evaluated the performance of athletes. However, due to the advancement of techniques, it may be difficult to accurately evaluate only by visual inspection of judges in some cases.

[0003] For this reason, conventional techniques for automatically recognizing the techniques of athletes are used. Hereinafter, an example of the conventional technique will be described. FIG. 15 is a diagram (1) for explaining an example of the conventional technique. An apparatus that executes the processing of the conventional technique is referred to as a "conventional apparatus".

[0004] For example, the conventional apparatus measures the distance information of an athlete using a 3D sensor, and generates a time-series skeleton frame based on the measurement result. For example, three-dimensional coordinates of each joint of the athlete are set in each skeleton frame. The conventional apparatus extracts a first feature amount from each skeleton frame. The first feature amount includes position information of body parts of the athlete, position information of joints, information on joint angles, and the like.

[0005] In FIG. 15, the explanation will be made using time-series skeleton frames f1-1 to f1-16. The conventional apparatus detects the state of the skeleton frame based on the first feature amounts of the skeleton frames f1-1 to f1-16. The state of the skeleton frame corresponds to the posture of the athlete shown in the skeleton frame. The relationship between the first feature amount and the state is defined in advance. When there is no state corresponding to the first feature amount, the state corresponding to the skeleton frame is "none".

[0006] For example, the states of the skeleton frames f1-1 to f1-3 are set to "standing upright". The states of the skeleton frames f1-9 to f1-11 are set to "inverted". The states of the skeleton frames f1-15 to f1-16 are set to "standing upright". The states of the other skeleton frames are set to "none".

[0007] Conventional devices detect segmental skeletal frames based on the detection results of the states of each skeletal frame f1-1 to f1-16. For example, conventional devices detect skeletal frames in a specific state as segmental frames. If skeletal frames in a specific state are consecutive, conventional devices use predetermined conditions to detect segmental skeletal frames. In the explanation of Figure 15, the specific states are referred to as "upright" and "inverted."

[0008] Conventional devices detect the skeletal frame with the spine pointing furthest upward as a segment when there is a sequence of skeletal frames in an "upright" state. For example, among skeletal frames f1-1 to f1-3 in an "upright" state, if skeletal frame f1-2 has the spine pointing furthest upward, the conventional device detects skeletal frame f1-2 as a segment. Similarly, among skeletal frames f1-15 to f1-16 in an "upright" state, if skeletal frame f15 has the spine pointing furthest upward, the conventional device detects skeletal frame f1-15 as a segment.

[0009] Conventional devices detect the skeletal frame with the spine pointing furthest downward as a segment when there is a series of skeletal frames in an "inverted" state. For example, in skeletal frames f1-9 to f1-11 in an "inverted" state, if skeletal frame f1-10 is the one with the spine pointing furthest downward, the conventional device detects skeletal frame f1-10 as a segment.

[0010] Conventional devices detect skeletal frames f1-2, f1-10, and f1-15 as segmentation points by performing the above process.

[0011] Conventional devices, after detecting segmentation points, classify each skeletal frame from the nth segmentation point to the (n+1)th segmentation point into the same group (where n is a natural number). Conventional devices calculate a second feature of a group based on the first feature included in the skeletal frames of the same group. In the explanation of Figure 15, the second feature is assumed to include forward posture, backward posture, somersault, twist, and highest point. Conventional devices identify the basic motion corresponding to the group based on the second feature and the conditions for the basic motion features. The conditions for the basic motion features are set in advance.

[0012] For example, a conventional device classifies each skeletal frame f1-2 to f1-10 contained between the first and second segmental points into group G1. A conventional device classifies each skeletal frame f1-10 to f1-15 contained between the second and third segmental points into group G2.

[0013] Conventional devices compare the second feature calculated from the first feature of skeletal frames f1-2 to f1-10 classified as group G1 with the feature conditions for each basic movement to identify a basic movement that satisfies the feature conditions. For example, a conventional device determines that the basic movement of group G1 is "two-foot takeoff from upright to back handstand" if the second feature of group G1 satisfies the feature conditions for the basic movement "two-foot takeoff from upright to back handstand". For example, the feature conditions for the basic movement "two-foot takeoff from upright to back handstand" are: forward position "upright", backward position "handstand", somersault "180°±90°", twist "0°±90°", and highest point "15cm or more".

[0014] Next, the conventional device compares the second feature calculated from the first feature of the skeletal frames f1-10 to f1-15 classified as group G2 with the feature conditions for each basic movement to identify the basic movement that satisfies the feature conditions. For example, the conventional device determines that the basic movement of group G2 is "handstand to back roll to upright" if the second feature of group G2 satisfies the feature conditions for the basic movement "handstand to back roll to upright". For example, the feature conditions for the basic movement "handstand to back roll to upright" are: forward position "handstand", backward position "upright", somersault "180°±90°", twist "0°±90°", and highest point "15cm or more".

[0015] The conventional device, by performing the above process, sequentially identifies the basic movement of Group G1, "two-footed takeoff from upright to backward roll to handstand," and the basic movement of Group G2, "handstand to backward roll to upright." The conventional device then determines the technique "backward roll jump" that corresponds to the combination of the basic movement "two-footed takeoff from upright to backward roll to handstand" and the basic movement "handstand to backward roll to upright."

[0016] Figure 16 is Figure (2) illustrating an example of the prior art. In the example shown in Figure 16, the explanation is given using time-series skeletal frames f1-1 to f1-26. The conventional device detects the state of the skeletal frames based on the first feature quantities of the skeletal frames f1-1 to f1-26.

[0017] For example, the state of skeletal frames f1-1 to f1-6 is defined as "downward rotation". The state of skeletal frames f1-21 to f1-26 is defined as "downward rotation". The state of all other skeletal frames is defined as "none".

[0018] Conventional devices detect segmental skeletal frames based on the detection results of the state of each skeletal frame f1-1 to f1-26. Conventional devices detect skeletal frames in a specific state as segmental frames. If skeletal frames in a specific state are consecutive, conventional devices use predetermined conditions to detect segmental skeletal frames. In the explanation of Figure 16, the specific state is defined as "downward rotation".

[0019] Conventional devices, when there is a sequence of skeletal frames in the "downward rotation" state, use the first skeletal frame on which both hands touch the floor as the segmentation point. For example, in skeletal frames f1-1 to f1-6, where the state is "downward rotation," if the first skeletal frame on which both hands touch the floor is frame f1-2, the conventional device detects frame f1-2 as the segmentation point. Similarly, in skeletal frames f1-21 to f1-26, where the state is "downward rotation," if the first skeletal frame on which both hands touch the floor is frame f1-23, the conventional device detects frame f1-23 as the segmentation point.

[0020] Conventional devices detect skeletal frames f1-2 and f1-23 as segmentation points by performing the above process.

[0021] Conventional devices, after detecting segmentation points, classify each skeletal frame from the nth segment to the (n+1)th segment into the same group. Conventional devices calculate a second feature of a group based on the first feature included in the skeletal frames of the same group. In the explanation of Figure 16, the second feature is assumed to include forward posture, backward posture, rotation, twist, and leg spread angle. Conventional devices identify basic movements based on the second feature and the conditions for the basic movement features. The conditions for the basic movement features are set in advance.

[0022] For example, conventional devices classify each skeletal frame f1-2 to f1-23 contained between the first and second segmental points into group G1.

[0023] The conventional device compares the second feature amount calculated from the first feature amounts of the skeleton frames f1-2 to f1-23 classified into the group G1 with the conditions of the feature amounts of each basic movement, and identifies the basic movement that satisfies the conditions of the feature amounts. For example, when the second feature amount of the group G1 satisfies the conditions of the feature amounts of the basic movement "one and a half turns with feet apart and a twist", the conventional device determines that the basic movement of the group G1 is "one and a half turns with feet apart and a twist". For example, the conditions of the feature amounts of the basic movement "one and a half turns with feet apart and a twist" are the front posture "downward turn", the rear posture "downward turn", the turn "360° ± 90°", the twist "180° ± 90°", and the foot-opening angle "60° or more".

[0024] By executing the above processing, the conventional device identifies the basic movement "one and a half turns with feet apart and a twist" of the group G1. The conventional device determines the technique "turning and twisting with feet apart" corresponding to the basic movement "one and a half turns with feet apart and a twist".

[0025] As described in FIGS. 15 and 16, in the prior art, based on the second feature amount of the group and the conditions of the feature amounts of each basic movement, the range of the skeleton frames corresponding to the basic movement is set, and the technique is determined from the combination of the basic movements.

Prior Art Documents

Patent Documents

[0026]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0027] However, in the above-described prior art, there is a problem that the technique cannot be correctly determined. For example, in the prior art, the range of the basic movement corresponding to the basic movement is set using the conditions of the feature amounts of the basic movement, and the determination accuracy of the technique may decrease.

[0028] In one aspect, the present invention aims to provide a judgment method, a judgment program, and an information processing device that can improve the accuracy of technique judgment. [Means for solving the problem]

[0029] In the first proposal, the computer performs the following processes: The computer detects multiple segments based on multiple frames containing a first feature that represents the characteristics of the subject's joints. The computer merges two or more of the multiple segments. The computer adjusts the segments to be merged so that a second feature identified from the first features of multiple frames included in the interval of the merged segments satisfies the conditions for a predetermined basic motion feature. [Effects of the Invention]

[0030] This can improve the accuracy of skill hit detection. [Brief explanation of the drawing]

[0031] [Figure 1] Figure 1 is a diagram illustrating the problems with the conventional technology. [Figure 2] Figure 2 is a diagram illustrating a simple solution to the prior art. [Figure 3] Figure 3 shows the system according to this embodiment. [Figure 4] Figure 4 is a diagram (1) illustrating the processing of the information processing apparatus according to this embodiment. [Figure 5] Figure 5 is a diagram (2) illustrating the processing of the information processing device according to this embodiment. [Figure 6] Figure 6 is a functional block diagram showing the configuration of the information processing device according to this embodiment. [Figure 7] Figure 7 shows an example of a human body model. [Figure 8] Figure 8 shows an example of joint names. [Figure 9] Figure 9 shows an example of the data structure of a segment definition table. [Figure 10] Figure 10 shows an example of the data structure of the basic movement definition table. [Figure 11] Figure 11 shows an example of the data structure of a technique definition table. [Figure 12] Figure 12 is a flowchart showing the processing procedure of the information processing device according to this embodiment. [Figure 13] Figure 13 shows an example of another embodiment. [Figure 14] Figure 14 shows an example of a computer hardware configuration that achieves similar functions to the information processing device in the embodiment. [Figure 15] Figure 15 is Figure (1) illustrating an example of the prior art. [Figure 16] Figure 16 is Figure (2) illustrating an example of the prior art. [Modes for carrying out the invention]

[0032] The following describes in detail, with reference to the drawings, embodiments of the determination method, determination program, and information processing device disclosed in this application. However, this embodiment does not limit the present invention. [Examples]

[0033] Before describing this embodiment, we will explain the problems of the prior art in more detail. Figure 1 is a diagram illustrating the problems of the prior art. The conventional device identifies the state of each skeletal frame based on the first feature of the time-series skeletal frame 10. The conventional device detects segmentation points based on the state of each skeletal frame. In Figure 1, as an example, the states of skeletal frames f2-1, f2-2, and f2-3 are each set to "downward rotation," and skeletal frames f2-1 to f2-3 are set as segmentation points. Note that the illustration of multiple skeletal frames included in skeletal frames f2-1 to f2-2 and f2-2 to f2-3 is omitted. In this embodiment, dividing the time-series skeletal frames is referred to as "segmentation," and the frames that are subject to segmentation are referred to as "skeletal frames that become segmentation points."

[0034] Conventional devices, after detecting segmentation points, classify each skeletal frame from the nth segment to the (n+1)th segment into the same group. Conventional devices calculate a second feature of a group based on the first feature included in the skeletal frames of the same group. In the explanation of Figure 1, the second feature is assumed to include forward posture, backward posture, rotation, twist, and leg spread angle. Conventional devices identify basic movements based on the second feature and the conditions for the basic movement features. The conditions for the basic movement features are set in advance.

[0035] Conventional devices classify each skeletal frame f2-1 to f2-2 contained between the first and second segmental points into group G1. Conventional devices classify each skeletal frame f2-2 to f2-3 contained between the second and third segmental points into group G2.

[0036] Conventional devices compare the second feature calculated from the first feature of skeletal frames f2-1 to f2-2 classified as group G1 with the feature conditions for each basic movement to identify a basic movement that satisfies the feature conditions. For example, a conventional device determines that the basic movement of group G1 is "one rotation and half twist with legs spread" if the second feature of group G1 satisfies the feature conditions for the basic movement "one rotation and half twist with legs spread". For example, the feature conditions for the basic movement "one rotation and half twist with legs spread" are: front posture "downward rotation", rear posture "downward rotation", rotation "360°±90°", twist "180°±90°", and leg spread angle "60° or more".

[0037] Next, the conventional device compares the second feature calculated from the first feature of the skeletal frames f2-2 to f2-3 classified as group G2 with the feature conditions for each basic movement to identify the basic movement that satisfies the feature conditions. For example, the conventional device determines that the basic movement of group G2 is "one rotation and half twist with legs spread" if the second feature of group G2 satisfies the feature conditions for the basic movement "one rotation and half twist with legs spread".

[0038] The conventional device, by performing the above process, sequentially identifies the basic movement of Group G1, "one turn and half twist with legs spread," and the basic movement of Group G2, "one turn and half twist with legs spread." The conventional device determines the technique "one turn and one twist with legs spread (with two turns)" in combination with the basic movement "one turn and half twist with legs spread."

[0039] Here, according to the gymnastics rules, the rotation (rotation angle) condition for the technique "straddle turn with one twist (with two rotations)" is "rotation = 630° to 810°". However, in conventional techniques, the technique "straddle turn with one twist (with two rotations)" is determined from a combination of basic movements, and not from the characteristic quantity conditions of the technique itself. For this reason, even if the rotation in skeletal frame 2-1 to f2-3 does not meet the condition of "630° to 810°", it may be incorrectly determined that the technique "straddle turn with one twist (with two rotations)" has been performed.

[0040] As shown in Figure 1, the rotational condition for the basic exercise "one turn with a 1 / 2 twist in a straddle position" is "360° ± 90°". For example, if the rotation of group G1 is "270°" and the rotation of group G2 is "270°", the total rotation will be "540°", which does not satisfy the rotational condition of "rotation = 630° to 810°" in the Gymnastics Rules. Similarly, if the rotation of group G1 is "450°" and the rotation of group G2 is "450°", the total rotation will be "900°", which does not satisfy the rotational condition of "rotation = 630° to 810°" in the Gymnastics Rules.

[0041] However, as mentioned above, with conventional equipment, there is a risk of incorrectly determining that the technique "straddle turn with one twist (with two turns)" has been performed if the rotation falls within the "540° to 900°" range. For example, according to the gymnastics rules, even if the rotation is "810°" or more and there is one twist, the technique "straddle turn with one twist (with two turns)" will not be recognized, but with conventional equipment, the technique "straddle turn with one twist (with two turns)" will incorrectly be determined to have been performed.

[0042] Furthermore, a simple solution to the problems of the conventional technology shown in Figure 1 is to treat multiple basic movements as a single basic movement when it is not possible to correctly judge a technique based on a combination of multiple basic movements.

[0043] Figure 2 illustrates a simple solution to the prior art. For the sake of explanation, the device that performs the processing shown in Figure 2 is referred to as the "reference device." The reference device identifies the state of each skeletal frame based on the first feature of the time-series skeletal frame 10. The reference device detects the segmentation points based on the state of each skeletal frame. In Figure 2, as an example, the states of skeletal frames f2-1, f2-2, and f2-3 are set to "downward rotation," and skeletal frames f2-1 to f2-3 are designated as segmentation points. Note that the illustration of multiple skeletal frames included in skeletal frames f2-1 to f2-2 and f2-2 to f2-3 is omitted.

[0044] The reference device, after detecting the segmentation points, classifies the skeletal frames f2-1 to f2-3 from the first to the last segmentation point into the same group G1. The reference device calculates the second feature quantity of group G1 based on the first feature quantity included in the skeletal frames of group G1. In the explanation of Figure 2, the second feature quantity is assumed to include forward posture, backward posture, rotation, twist, and leg spread angle. The reference device identifies the basic motion based on the second feature quantity and the conditions for the basic motion feature quantity. The conditions for the basic motion feature quantity are set in advance.

[0045] The reference device compares the second feature calculated from the first feature of skeletal frames f2-1 to f2-3 classified as group G1 with the feature conditions for each basic movement to identify a basic movement that satisfies the feature conditions. For example, the reference device determines that the basic movement of group G1 is "two rotations and one twist with legs spread" if the second feature of group G1 satisfies the feature conditions for the basic movement "two rotations and one twist with legs spread". For example, the feature conditions for the basic movement "two rotations and one twist with legs spread" are: front posture "downward rotation", rear posture "downward rotation", rotation "720°±90°", twist "360°±90°", and leg spread angle "60° or more".

[0046] The reference device identifies the basic movement of Group G1, "two turns and one twist in a straddle position," by performing the above process. The reference device determines the technique that corresponds to the basic movement "two turns and one twist in a straddle position," which is "one twist in a straddle position (with two turns)."

[0047] According to the reference device explained in Figure 2, the rotation conditions for the basic movement "two turns and one twist with legs spread" match the conditions for the technique "one twist with legs spread (with two turns)," thus allowing for correct technique recognition, unlike conventional techniques. However, a problem arises in that it cannot identify each basic movement "one turn and half twist with legs spread," which was distinguishable with conventional techniques.

[0048] Next, this embodiment will be described. Figure 3 is a diagram showing the system according to this embodiment. As shown in Figure 3, this system 30 has cameras 31a, 31b, 31c, and 31d, and an information processing device 100. The cameras 31a to 31d are connected to the information processing device 100 by wire or wireless.

[0049] Cameras 31a to 31d are each installed in different positions to capture images of the players (RGB).<Red Green Blue> Cameras 31a to 31d capture an image. Cameras 31a to 31d transmit the captured image data to the information processing device 100. The image data captured by cameras 31a to 31d is referred to as an "image frame". Cameras 31a to 31d transmit multiple image frames in chronological order to the information processing device 100. Each image frame is assigned a frame number in ascending order. In the following explanation, cameras 31a to 31d will be collectively referred to as "camera 31" as appropriate.

[0050] The information processing device 100 has a trained skeletal inference model and generates a time-series skeletal frame by inputting time-series image frames acquired from the camera 31 into the skeletal inference model. For example, the skeletal frame contains the 3D coordinates of each joint of the athlete. Based on the time-series skeletal frame, the information processing device 100 performs the following processing to determine the athlete's technique.

[0051] Figure 4 is a diagram (1) illustrating the processing of the information processing device according to this embodiment. The information processing device 100 extracts a first feature from the time-series skeletal frame 15. The first feature includes positional information of body parts of the athlete, joint positional information, joint angle information, etc. Based on the first feature of the time-series skeletal frame 15, the information processing device 100 identifies the state of each skeletal frame. Based on the state of each skeletal frame, the information processing device 100 detects the segmentation points. In Figure 4, as an example, the states of skeletal frames f3-1, f3-2, f3-3, and f3-4 are each set to "downward rotation," and skeletal frames f3-1 to f3-4 are set as segmentation points. In Figure 4, the illustration of the skeletal frames included in skeletal frames f3-1 to f3-2, f3-2 to f3-3, and f3-3 to f3-4 is omitted.

[0052] After detecting a segmentation point, the information processing device 100 merges two or more segmentation points and classifies multiple skeletal frames contained between the starting segmentation point and the ending segmentation point into the same group. The information processing device 100 calculates a second feature of a group based on the first feature of each skeletal frame contained in the same group. If the second feature of a group satisfies the conditions of any of the basic motion features, the information processing device 100 associates the multiple skeletal frames contained in the current group with the basic motion. On the other hand, if the second feature of a group does not satisfy the conditions of any of the basic motion features, the information processing device 100 adjusts the segmentation points to be merged and repeats the above process.

[0053] If the state of the skeletal frame detected as each segment is the same, the information processing device 100 adjusts the segment points so that it divides sequentially from the last segment point, using the first segment point as the reference. On the other hand, if the state of the skeletal frame detected as each segment is not the same, the information processing device 100 adjusts the segment points so that it divides sequentially from the first segment point, using the last segment point as the reference.

[0054] In the example shown in Figure 4, the state of the skeletal frame detected as each segment is the same, "rotating downwards." Therefore, the information processing device 100 adjusts the segment points so that it divides sequentially from the later segment points, using the first segment point as the reference point.

[0055] The first processing of the information processing device 100 will be explained using Figure 4. The information processing device 100 integrates the first segment (frame f3-1) through the fourth segment (frame f3-4) and classifies the skeletal frames included in frames f3-1 to f3-4 into the same group G1. The information processing device 100 calculates the second feature quantity of group G1 based on the first feature quantity of the skeletal frames included in group G1. For example, the second feature quantity of group G1 may be rotation "1080° (3 rotations)" and twist "360° (1 twist)".

[0056] The information processing device 100 compares the second feature of group G1 with the conditions for the features of each basic motion to determine whether the second feature of group G1 satisfies the conditions for any of the basic motion features. Here, we will continue the explanation assuming that the second feature of group G1 does not satisfy the conditions for any of the basic motion features.

[0057] The second processing of the information processing device 100 is described below. The information processing device 100 adjusts the segment to be merged because the second feature of group G1 does not satisfy the conditions for any of the basic motion feature quantities. The information processing device 100 merges the first segment (frame f3-1) to the third segment (frame f3-3) and classifies the skeletal frames included in frames f3-1 to f3-3 into the same group G2. The information processing device 100 calculates the second feature of group G2 based on the first feature of the skeletal frames included in group G2. For example, the second feature of group G2 is set to rotation "720° (2 rotations)" and twist "180° (1 / 2 twist)".

[0058] The information processing device 100 compares the second feature of group G2 with the conditions for the features of each basic motion to determine whether the second feature of group G2 satisfies the conditions for any of the basic motion features. Here, we will continue the explanation assuming that the second feature of group G2 does not satisfy the conditions for any of the basic motion features.

[0059] The third processing step of the information processing device 100 is described below. The information processing device 100 adjusts the segment to be merged because the second feature of group G2 does not satisfy the conditions for any of the basic motion feature quantities. The information processing device 100 merges the second segment (frame f3-2) from the first segment (frame f3-1) and classifies the skeletal frames included in frames f3-1 to f3-2 into the same group G3. The information processing device 100 calculates the second feature of group G3 based on the first feature of the skeletal frames included in group G3. For example, the second feature of group G3 is set to rotation "360° (1 rotation)" and twist "none".

[0060] The information processing device 100 compares the second feature of group G3 with the conditions for the feature of each basic movement to determine whether the second feature of group G3 satisfies the conditions for any of the basic movement features. Here, it is assumed that the feature of group G3 satisfies the conditions for the feature of the basic movement "split-leg rotation". Based on this, the information processing device 100 identifies the basic movement "split-leg rotation" that corresponds to group G3.

[0061] The fourth processing step of the information processing device 100 is described below. The information processing device 100 continues processing the skeletal frames f3-2 to f3-4, excluding skeletal frames f3-1 to f3-2, which are classified as group G3. For example, the information processing device 100 integrates the second segment (frame f3-2) to the fourth segment (frame f3-4), and classifies the skeletal frames included in frames f3-2 to f3-4 into the same group G4. The information processing device 100 calculates the second feature quantity of group G4 based on the first feature quantity of the skeletal frames included in group G4. For example, the second feature quantity of group G4 is set to rotation "720° (2 rotations)" and twist "360° (1 twist)".

[0062] The information processing device 100 compares the second feature of group G4 with the conditions for the feature of each basic movement to determine whether the second feature of group G4 satisfies the conditions for any of the basic movement's feature conditions. Here, it is assumed that the feature of group G4 satisfies the conditions for the feature of the basic movement "split leg, two rotations, one twist". Based on this, the information processing device 100 identifies the basic movement "split leg, two rotations, one twist" that corresponds to group G4.

[0063] As explained in Figure 4, the information processing device 100 identifies the basic movement "straddle turn" and the basic movement "straddle turn with one twist" by performing the first to fourth processing steps. The information processing device 100 also determines the technique "straddle turn" and "straddle turn with one twist (with two turns)" which are combinations of the basic movement "straddle turn" and the basic movement "straddle double turn with one twist". Note that although Figure 4 explains the case where the number of segmentation points is "4", the number of segmentation points is not limited to 4.

[0064] Figure 5 is a diagram (2) illustrating the processing of the information processing device according to this embodiment. The information processing device 100 extracts a first feature from the time-series skeletal frame 16. Based on the first feature of the time-series skeletal frame 16, the information processing device 100 identifies the state of each skeletal frame. Based on the state of each skeletal frame, the information processing device 100 detects the segmentation point. In Figure 5, as an example, the states of skeletal frames f4-1, f4-2, and f4-3 are set to "downward rotation," and the state of skeletal frame f4-4 is set to "inverted."

[0065] In the example shown in Figure 5, the states of the skeletal frame detected as each segment are "downward rotation" and "inverted," so the states are not identical. Therefore, the information processing device 100 adjusts the segments so that they are divided sequentially from the previous segment, using the last segment as the reference point.

[0066] The first processing of the information processing device 100 will be explained using Figure 5. The information processing device 100 integrates the first segment (frame f4-1) through the fourth segment (frame f4-4) and classifies the skeletal frames included in frames f4-1 to f4-4 into the same group G1. The information processing device 100 calculates the second feature of group G1 based on the first feature of the skeletal frames included in group G1. For example, the second feature of group G1 is set to an inversion from a rotation of "1080° (3 rotations)" and a twist of "270° (3 / 4 twist)".

[0067] The information processing device 100 compares the second feature of group G1 with the conditions for the features of each basic motion to determine whether the second feature of group G1 satisfies the conditions for any of the basic motion features. Here, we will continue the explanation assuming that the second feature of group G1 does not satisfy the conditions for any of the basic motion features.

[0068] The second processing of the information processing device 100 is described below. The information processing device 100 adjusts the segment to be merged because the second feature of group G1 does not satisfy the conditions for any of the basic motion feature quantities. The information processing device 100 merges the second segment (frame f4-2) to the fourth segment (frame f4-4) and classifies the skeletal frames included in frames f4-2 to f4-4 into the same group G2. The information processing device 100 calculates the second feature of group G2 based on the first feature of the skeletal frames included in group G2. For example, the second feature of group G2 is set to rotation "720° (2 rotations)" and twist "270° (3 / 4 twist)".

[0069] The information processing device 100 compares the second feature of group G2 with the conditions for the feature of each basic movement to determine whether the second feature of group G2 satisfies the conditions for any of the basic movement's feature conditions. Here, it is assumed that the feature of group G2 satisfies the conditions for the feature of the basic movement "two turns with legs spread, twist of 270° or more, and a direct handstand". Based on this, the information processing device 100 identifies the basic movement "two turns with legs spread, twist of 270° or more, and a direct handstand" that corresponds to group G2.

[0070] The third processing step of the information processing device 100 is described below. The information processing device 100 continues processing the skeletal frames f4-1 to f4-2, excluding the skeletal frames f4-2 to f4-4 that are classified as group G2. For example, the information processing device 100 merges the second segment (frame f4-2) from the first segment (frame f4-1) and classifies the skeletal frames included in frames f4-1 to f4-2 into the same group G3. The information processing device 100 calculates the second feature quantity of group G3 based on the first feature quantity of the skeletal frames included in group G3. For example, the second feature quantity of group G3 is set to rotation "360° (1 rotation)" and no twist.

[0071] The information processing device 100 compares the second feature of group G3 with the conditions for the feature of each basic movement to determine whether the second feature of group G3 satisfies the conditions for any of the basic movement features. Here, it is assumed that the feature of group G3 satisfies the conditions for the feature of the basic movement "split-leg rotation". Based on this, the information processing device 100 identifies the basic movement "split-leg rotation" that corresponds to group G3.

[0072] As explained in Figure 5, the information processing device 100 identifies the basic movement "split-legged turn" and the basic movement "split-legged double turn with a twist of 270° or more followed by a direct handstand" by performing the first to third processing steps. The information processing device 100 also determines the techniques "split-legged turn" and "split-legged turn with a twist of 270° or more followed by a direct handstand (with two turns)" that correspond to the combination of the basic movement "split-legged turn" and the basic movement "split-legged double turn with a twist of 270° or more followed by a direct handstand".

[0073] As described above, the information processing device 100 according to this embodiment integrates two or more segmental points and classifies multiple skeletal frames included from the starting segmental point to the ending segmental point into the same group at the integrated segmental point. If the second feature of the same group satisfies the conditions of any of the basic motion feature quantities, the information processing device 100 associates the multiple skeletal frames included in the current group with the basic motion. On the other hand, if the second feature of the group does not satisfy the conditions of any of the basic motion feature quantities, the information processing device 100 adjusts the segmental points to be integrated and repeatedly performs the above process.

[0074] This allows for the recognition of the largest unit of basic movement, and by utilizing the results of this basic movement recognition, the accuracy of technique judgment can be improved. For example, techniques that cannot be accurately recognized by combinations of basic movements in intervals divided by the smallest unit of segmentation can be accurately recognized.

[0075] Next, an example configuration of the information processing device 100 that performs the processes described in Figures 4 and 5 will be described. Figure 6 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Figure 6, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0076] The communication unit 110 performs data communication with the camera 31, external devices, etc., via the network. The communication unit 110 is a NIC (Network Interface Card), etc. For example, the communication unit 110 receives time-series image frames from the camera 31.

[0077] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100. For example, the input unit 120 can be a keyboard, mouse, touch panel, etc.

[0078] The display unit 130 is a display device that displays information output from the control unit 150.

[0079] The memory unit 140 includes a skeletal inference model 141, a segment definition table 142, a basic movement definition table 143, and a technique definition table 144. The memory unit 140 is a storage device such as a memory.

[0080] The skeletal inference model 141 is a model that, when given an image frame captured by camera 31 as input, outputs the skeletal frame of the player contained in the image frame. The skeletal inference model 141 is a Neural Network (NN) or similar and is assumed to be pre-trained.

[0081] The skeletal frame is information in which three-dimensional coordinates are assigned to multiple joints defined in the human body model. Figure 7 shows an example of a human body model. As shown in Figure 7, the human body model is defined by 21 joints ar0 to ar20.

[0082] The relationship between each joint ar0 to ar20 shown in Figure 7 and its name is shown in Figure 8. Figure 8 is a diagram showing an example of a joint name. For example, the name of joint ar0 is "SPINE_BASE". The names of joints ar1 to ar20 are as shown in Figure 8, and their explanation is omitted.

[0083] Let's return to the explanation of Figure 6. The segment definition table 142 is a table that defines the conditions for the feature quantities (first features) of the skeletal frames to be detected as segments. Figure 9 shows an example of the data structure of the segment definition table. As shown in Figure 9, the segment definition table 142 associates states with the conditions for the feature quantities (first features). The states correspond to the posture of the players.

[0084] The basic motion definition table 143 is a table that defines the conditions for the features (secondary features) of basic motions. Figure 10 shows an example of the data structure of the basic motion definition table. As shown in Figure 10, the basic motion definition table 143 associates basic motions with the conditions for the features (secondary features).

[0085] For example, the feature conditions for the basic movement "split leg, one rotation and 1 / 2 twist" are: front posture "downward rotation", back posture "downward rotation", rotation "360°±90°", twist "180°±90°", and leg split angle "60° or more". Here, the front posture indicates the state of the starting segment among the multiple integrated segmental points. The back posture indicates the state of the ending segment among the multiple integrated segmental points. The rotation is the rotation angle based on the athlete's spine vector. For example, the spine vector is the vector from joint ar0 to joint ar2 in the human body model in Figure 7. The twist is the twist angle based on the athlete's spine vector. The leg split angle is the angle between the athlete's right foot vector and left foot vector. The right foot vector is the vector from joint ar14 to joint ar16 in the human body model in Figure 7. The left foot vector is the vector pointing from joint ar10 to joint ar12 in the human body model in Figure 7.

[0086] The characteristic features of the basic movement "two turns and one twist with legs spread" are: forward position "downward turn", backward position "downward turn", turn "720°±90°", twist "360°±90°", and leg spread angle "60° or more".

[0087] Although not shown in the diagram, the basic movement definition table 143 sets feature conditional data for the basic movements "split leg rotation," "split leg rotation with one twist," and other basic movements.

[0088] The technique definition table 144 is a table that defines the relationship between basic movements (or combinations of basic movements) and techniques defined in the Gymnastics Rules. Figure 11 shows an example of the data structure of the technique definition table. As shown in Figure 11, the technique definition table 144 associates basic movements with techniques. Even for the same athlete's movements, the name of the basic movement and the name of the technique may differ. For example, the technique corresponding to the basic movement "straddle turn" is "straddle turn". The technique corresponding to the basic movement "straddle double turn with one twist" is "straddle turn with one twist (with two turns)".

[0089] We will now move on to the explanation of the control unit 150 in Figure 6. The control unit 150 includes an acquisition unit 151, a skeletal frame generation unit 152, a first feature calculation unit 153, a segment detection unit 154, a segment adjustment unit 155, a second feature calculation unit 156, a basic motion identification unit 157, and a technique judgment unit 158. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0090] The acquisition unit 151 acquires time-series image frames from the camera 31. The acquisition unit 151 outputs the image frames to the skeletal frame generation unit 152.

[0091] The skeletal frame generation unit 152 generates a time-series skeletal frame by inputting time-series image frames into the skeletal inference model 141. The skeletal frame generation unit 152 outputs the time-series skeletal frame to the first feature calculation unit 153. The skeletal frame generation unit 152 may assign frame numbers sequentially to the time-series skeletal frames.

[0092] The first feature calculation unit 153 calculates a first feature for each skeletal frame in the time series. That is, the first feature calculation unit 153 calculates one first feature from each skeletal frame. The first feature includes positional information of each part of the athlete's body, positional information of each joint, joint angle information, etc.

[0093] The first feature calculation unit 153 outputs information that associates the time-series skeletal frame with the first feature to the segment detection unit 154.

[0094] The segment detection unit 154 detects skeletal frames that will become segments based on the first feature of the time-series skeletal frame and the segment definition table 142. For example, the segment detection unit 154 compares the feature conditions of the segment definition table 142 with the first feature of each skeletal frame and detects the skeletal frame corresponding to the first feature that satisfies the feature conditions as a segment. When the segment detection unit 154 detects a segment, it also determines the state corresponding to the segment.

[0095] The segmental point detection unit 154 detects a segmental skeletal frame using predetermined conditions when there are consecutive skeletal frames that serve as segmental points. For example, if there are consecutive skeletal frames where the segmental point state is "downward rotation", the segmental point detection unit 154 detects the first skeletal frame in which both hands touch the floor as a segmental point. For example, the segmental point detection unit 154 determines that both hands have touched the floor when the z (height) of the 3D coordinates (x, y, z) of joints ar19 and ar20 in Figure 7 is less than a threshold.

[0096] The segment detection unit 154 outputs the first feature of the time-series skeleton frame and the segment information to the segment adjustment unit 155. The segment information is associated with the frame number of the skeleton frame that serves as the segment and the state of the skeleton frame.

[0097] The segment adjustment unit 155 merges two or more segments and classifies each skeletal frame included from the starting segment to the ending segment of the merged segment into the same group. The segment adjustment unit 155 outputs the first feature of each skeletal frame included in the same group to the second feature calculation unit 156.

[0098] If the segment adjustment unit 155 obtains information from the basic motion identification unit 157 indicating that the second feature of a group satisfies the conditions of any of the basic motion features, it determines the integrated segment and repeatedly performs the above process on the unprocessed segment.

[0099] If the segment adjustment unit 155 obtains information from the basic motion identification unit 157 indicating that the second feature of the group does not satisfy the conditions of any of the basic motion features, it adjusts the integrated segment and repeatedly performs the above process.

[0100] Here, the process by which the segmentation point adjustment unit 155 adjusts the segments is the same as the process described in Figures 4 and 5. If the state of the skeletal frame detected as each segment is the same, the segmentation point adjustment unit 155 adjusts the segments so that they are divided sequentially from the later segments, based on the first segment, as described in Figure 4. For example, the segmentation point adjustment unit 155 performs an adjustment to exclude the endpoint segment from the merged segments.

[0101] On the other hand, if the state of the skeletal frame detected as each segment is not the same, the information processing device 100 adjusts the segments so that they are divided sequentially from the previous segment, using the last segment as the reference point, as explained in Figure 5. For example, the segment adjustment unit 155 performs an adjustment to exclude the starting segment from the segment to be integrated among the integrated segments.

[0102] The second feature calculation unit 156 calculates the second feature of a group based on the first feature of each skeletal frame included in the same group. For example, the second feature may include forward posture, backward posture, rotation, twist, leg spread angle, etc. The second feature calculation unit 156 sets the state of the starting segment (skeletal frame) among the skeletal frames included in the group to forward posture. The second feature calculation unit 156 sets the state of the ending segment (skeletal frame) among the skeletal frames included in the group to backward posture.

[0103] The second feature calculation unit 156 identifies the spine vector of each skeletal frame included in the group and calculates the change in angle of rotation of the spine vector from the starting point to the ending point as rotation (rotation angle).

[0104] The second feature calculation unit 156 identifies the spine vector of each skeletal frame included in the group and calculates the change in the angle of the spine vector's twist from the starting point to the ending point as the twist (twist angle).

[0105] The second feature calculation unit 156 identifies the right foot vector and left foot vector for each skeletal frame included in the group, and calculates the angle between the right foot vector and the left foot vector. For each skeletal frame, the second feature calculation unit 156 calculates the angle between the right foot vector and the left foot vector, and calculates the maximum value of the angle as the leg-spread angle.

[0106] The second feature calculation unit 156 calculates the second feature of the group by performing the above processing and outputs the second feature of the group to the basic motion identification unit 157. The second feature calculation unit 156 may calculate the second feature using other known techniques. In addition, the second feature calculation unit 156 may calculate features other than forward posture, backward posture, rotation, twist, and leg spread angle as second features.

[0107] The basic motion identification unit 157 identifies the basic motion corresponding to a group based on the group's second feature and the feature condition of the basic motion definition table 143. The basic motion identification unit 157 identifies the basic motion corresponding to the condition of any of the feature conditions in the basic motion definition table 143 if the group's second feature satisfies the condition of the feature of that feature. For example, if the group's second feature satisfies the feature condition of the basic motion "spread legs, one rotation, and a 1 / 2 twist," the basic motion identification unit 157 identifies that the basic motion shown for each skeletal frame of the group is "spread legs, one rotation, and a 1 / 2 twist."

[0108] On the other hand, the basic motion identification unit 157 determines that if the second feature of a group does not satisfy any of the feature conditions in the basic motion definition table 143, then there is no basic motion corresponding to the second feature of the group.

[0109] The basic movement identification unit 157 outputs the identification result to the segment adjustment unit 155. The basic movement identification unit 157 also outputs the information of the identified basic movement to the technique judgment unit 158.

[0110] The technique determination unit 158 ​​determines the technique based on the basic movement identified by the basic movement identification unit 157 and the technique definition table 144. The technique determination unit 158 ​​displays the technique determination result on the display unit 130.

[0111] Next, the processing procedure of the information processing device 100 according to this embodiment will be described. Figure 12 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Figure 12, the acquisition unit 151 of the information processing device 100 acquires time-series image frames from the camera 31 (step S101). The time-series image frames of the information processing device 100 are input to the skeletal inference model 141 to generate a time-series skeletal frame (step S102).

[0112] The first feature calculation unit 153 of the information processing device 100 calculates the first feature of each skeletal frame (step S103). The segment detection unit 154 of the information processing device 100 detects segments based on the first feature of each skeletal frame and the segment definition table 142 (step S104).

[0113] The segmentation point adjustment unit 155 of the information processing device 100 integrates the segmentation points and classifies the multiple skeletal frames included from the start point to the end point of the segmentation point into the same group (step S105). The second feature calculation unit 156 of the information processing device 100 calculates the second feature of the group (step S106).

[0114] The basic motion identification unit 157 of the information processing device 100 proceeds to step S108 if, based on the basic motion definition table 143, the second feature of the group does not satisfy the conditions for any of the basic motion features (step S107, No). On the other hand, the basic motion identification unit 157 proceeds to step S109 if, based on the basic motion definition table 143, the second feature of the group satisfies the conditions for any of the basic motion features (step S107, Yes).

[0115] The segmentation point adjustment unit 155 adjusts the segmentation point and classifies the multiple skeletal frames included from the start point to the end point of the segmentation point into the same group (step S108), and then proceeds to step S106.

[0116] The basic motion identification unit 157 identifies the basic motion corresponding to the group (step S109). The segment adjustment unit 155 removes the skeletal frame of the group corresponding to the basic motion from the time-series skeletal frame (step S110).

[0117] If the skeletal frame to be processed has multiple segments, the segment adjustment unit 155 proceeds to step S105 (step S111, Yes). On the other hand, if the skeletal frame to be processed does not have multiple segments, the technique determination unit 158 ​​of the information processing device 100 determines the technique based on the technique definition table 144 (step S112).

[0118] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 integrates two or more segmental points and classifies multiple skeletal frames included from the starting segmental point to the ending segmental point into the same group at the integrated segmental point. If the second feature quantity of the same group satisfies the conditions of any of the basic motion feature quantities, the information processing device 100 associates the multiple skeletal frames included in the current group with the basic motion. On the other hand, if the second feature quantity of the group does not satisfy the conditions of any of the basic motion feature quantities, the information processing device 100 adjusts the segmental points to be integrated and repeatedly performs the above process.

[0119] This allows for the recognition of the largest unit of basic movement, and by utilizing the recognition results of such basic movement, the accuracy of technique judgment can be improved. For example, techniques that cannot be accurately recognized by combinations of basic movements in intervals divided by the smallest unit of segmentation can be accurately judged.

[0120] The information processing device 100 determines the technique based on the basic movements and the technique definition table 144. This allows for accurate determination of the athlete's technique.

[0121] When adjusting integrated segmental points, the information processing device 100 performs an adjustment to exclude the endpoint segment from the integration target if the orientations of the integrated segmental points are the same. Furthermore, if the orientations of the integrated segmental points are different, the information processing device 100 performs an adjustment to exclude the starting segment from the integration target. This allows for accurate recognition of the fundamental motion of the largest unit.

[0122] By the way, in this embodiment, as an example, the information processing device 100 acquired time-series image frames from the camera 31 and generated time-series skeletal frames, but it is not limited to this. For example, the information processing device 100 may use a 3D sensor to measure distance images of the players and generate time-series skeletal frames based on the measurement results.

[0123] Furthermore, in the processing of the information processing device 100 according to this embodiment, the focus was on gymnastics, and when the state of the skeletal frame entered a specific state such as "downward rotation," the skeletal frame was detected as a segment point, but the invention is not limited to this.

[0124] Figure 13 shows an example of another embodiment. The information processing device 100 can also detect segmental skeletal frames for breakdancers. For example, in a move called a windmill, the dancer repeatedly rotates in a position called a "chair". The information processing device 100 detects the skeletal frames corresponding to the chair position from the time-series skeletal frames as segmental points. The processing after the information processing device 100 detects the segmental points is the same as the processing described above.

[0125] Next, an example of a computer hardware configuration that achieves the same functions as the information processing device 100 described above will be explained. Figure 14 shows an example of a computer hardware configuration that achieves the same functions as the information processing device in the embodiment.

[0126] As shown in Figure 14, the computer 300 includes a CPU 301 that performs various calculations, an input device 302 that receives data input from the user, and a display 303. The computer 300 also includes a communication device 304 and an interface device 305 that exchange data with external devices via a wired or wireless network. Furthermore, the computer 300 includes a RAM 306 for temporarily storing various information and a hard disk drive 307. Each of these devices 301 to 307 is connected to a bus 308.

[0127] The hard disk drive 307 includes an acquisition program 307a, a skeletal frame generation program 307b, a first feature calculation program 307c, and a segment detection program 307d. The hard disk drive 307 also includes a segment adjustment program 307e, a second feature calculation program 307f, a basic motion recognition program 307g, and a technique judgment program 307h. The CPU 301 reads each program 307a to 307h and loads them into the RAM 306.

[0128] The acquisition program 307a functions as the acquisition process 306a. The skeletal frame generation program 307b functions as the skeletal frame generation process 306b. The first feature calculation program 307c functions as the first feature calculation process 306c. The segment detection program 307d functions as the segment detection process 306d. The segment adjustment program 307e functions as the segment adjustment process 306e. The second feature calculation program 307f functions as the second feature calculation process 306f. The basic motion identification program 307g functions as the basic motion identification process 306g. The technique judgment program 307h functions as the technique judgment process 306h.

[0129] The processing of acquisition process 306a corresponds to the processing of acquisition unit 151. The processing of skeletal frame generation process 306b corresponds to the processing of skeletal frame generation unit 152. The processing of first feature calculation process 306c corresponds to the processing of first feature calculation unit 153. The processing of segment detection process 306d corresponds to the processing of segment detection unit 154. The processing of segment adjustment process 306e corresponds to the processing of segment adjustment unit 155. The processing of second feature calculation process 306f corresponds to the processing of second feature calculation unit 156. The processing of basic motion identification process 306 corresponds to the processing of basic motion identification unit 157. The processing of technique judgment process 306h corresponds to the processing of technique judgment unit 158.

[0130] Furthermore, programs 307a to 307h do not necessarily have to be stored on the hard disk drive 307 from the beginning. For example, each program could be stored on a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 300. Then, the computer 300 could read and execute each program 307a to 307h. [Explanation of Symbols]

[0131] 100 Information Processing Devices 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Skeletal Inference Model 142 Segment Definition Table 143 Basic Exercise Definition Table 144 Technique Definition Table 150 Control Unit 151 Acquisition Department 152 Skeleton Frame Generation Unit 153 First Feature Calculation Unit 154 Segment detection unit 155. Sectional adjustment unit 156 Second Feature Calculation Unit 157 Basic motion identification unit 158 Skill Judgment Department

Claims

1. Based on multiple frames containing a first feature that represents the joint features of the subject, multiple segmental points are detected. Of the aforementioned multiple segmental points, two or more segmental points are merged, The segmental points to be merged are adjusted so that the second feature, identified from the first feature of multiple frames included in the interval of the merged segmental points, satisfies the conditions for a predetermined basic motion feature. A determination method characterized by the processing being performed by a computer.

2. The determination method according to claim 1, characterized in that the adjustment process adjusts the integrated segmental points so that the second feature satisfies the condition for a feature of basic motion including one or more turns.

3. The determination method according to claim 1, characterized in that the computer further performs a process to determine the technique based on the basic movements corresponding to the integrated segmental points.

4. The determination method according to claim 1, wherein the computer further performs a process to detect the posture of the subject based on the first feature quantity of the frame corresponding to the segment, and the adjustment process is characterized in that, if the postures of the integrated segment are the same, the adjustment excludes the endpoint segment from the integration target among the integrated segment.

5. The determination method according to claim 4, characterized in that the adjustment process, when the orientations of the integrated segmental points are different, performs an adjustment to exclude the starting segmental point from the integration target among the integrated segmental points.

6. Based on multiple frames containing a first feature that represents the joint features of the subject, multiple segmental points are detected. Of the aforementioned multiple segmental points, two or more segmental points are merged, The segmental points to be merged are adjusted so that the second feature, identified from the first feature of multiple frames included in the interval of the merged segmental points, satisfies the conditions for a predetermined basic motion feature. A determination program characterized by having a computer perform the processing.

7. Based on multiple frames containing a first feature that represents the joint features of the subject, multiple segmental points are detected. Of the aforementioned multiple segmental points, two or more segmental points are merged, The segmental points to be merged are adjusted so that the second feature, identified from the first feature of multiple frames included in the interval of the merged segmental points, satisfies the conditions for a predetermined basic motion feature. An information processing device having a control unit that performs processing.

Images