Information processing method, program, information processing system, and information processing terminal

Abstract

To provide an information processing method for enabling a user to efficiently improve a series of operations.SOLUTION: A first importance obtained by multiplying a first score by a first weight, the first score being obtained by evaluating, based on a moving image of a subject performing a series of motions, whether a posture of a first part of the subject in a first reference posture serving as a reference for the series of motions is better or worse than a posture of a first part of a person serving as a model in the first reference posture, the first weight being associated with the first part in the first reference posture; Calculating a second degree of importance obtained by multiplying a second score evaluating whether a posture of a second part of the person in a second reference posture is good or bad by a second weight associated with the second part in the second reference posture, the first weight indicating a degree of importance of improving a posture of a first part in a first reference posture, the second weight indicating a degree of importance of improving a posture of the second part in the second reference posture.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, a program, an information processing system, and an information processing terminal. [Background technology]

[0002] Patent document 1 discloses an animation generation system that calculates a score to evaluate the quality of each part of a user's body in a series of movements in a predetermined exercise based on data showing the user's movements and data showing the movements of a model person, and displays an animation to improve the movement of the part that performed the movement with the lowest calculated score. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-152784 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, improving the movement of the part of the body that performed the movement with the lowest score may not lead to an improvement in the entire series of movements performed by the user, and it may not be possible to efficiently improve the series of movements performed by the user. [Means for solving the problem]

[0005] In order to solve the above problem, one aspect of the present disclosure is an information processing method that calculates, based on a video image of a subject performing a series of movements, a first importance by multiplying a first score that evaluates whether the posture of a first part of the subject in a first reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the first part of a model person in the first reference posture by a first weight associated with the first part in the first reference posture, and a second importance by multiplying a second score that evaluates whether the posture of a second part of the subject in a second reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the second part of the model person in the second reference posture by a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture and the second weight indicates the importance of improving the posture of the second part in the second reference posture.

[0006] In order to solve the above problem, one aspect of the present disclosure is a program that causes a computer to calculate, based on a video image of a subject performing a series of movements, a first importance obtained by multiplying a first score that evaluates whether the posture of a first part of the subject in a first reference posture that serves as the basis for the series of movements is good or bad compared to the posture of the first part of a model person in the first reference posture by a first weight associated with the first part in the first reference posture; and a second importance obtained by multiplying a second score that evaluates whether the posture of a second part of the subject in a second reference posture that serves as the basis for the series of movements is good or bad compared to the posture of the second part of the model person in the second reference posture by a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture and the second weight indicates the importance of improving the posture of the second part in the second reference posture.

[0007] In order to solve the above problem, one aspect of the present disclosure is an information processing system that calculates, based on a video image of a subject performing a series of movements, a first importance obtained by multiplying a first score that evaluates whether the posture of a first part of the subject in a first reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the first part of a model person in the first reference posture by a first weight associated with the first part in the first reference posture; and a second importance obtained by multiplying a second score that evaluates whether the posture of a second part of the subject in a second reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the second part of the model person in the second reference posture by a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture and the second weight indicates the importance of improving the posture of the second part in the second reference posture.

[0008] In order to solve the above problem, one aspect of the present disclosure is an information processing terminal that calculates, based on a video image of a subject performing a series of movements, a first importance obtained by multiplying a first score that evaluates whether the posture of a first part of the subject in a first reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the first part of a model person in the first reference posture by a first weight associated with the first part in the first reference posture; and a second importance obtained by multiplying a second score that evaluates whether the posture of a second part of the subject in a second reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the second part of the model person in the second reference posture by a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture and the second weight indicates the importance of improving the posture of the second part in the second reference posture. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system 1. FIG. [Figure 2]This is a diagram showing the five standard positions in a golf swing arranged in chronological order. [Figure 3] 10A to 10C are diagrams showing examples of displaying a moving image of an entire subject's swing, a moving image of a partial subject's swing, and a moving image of a target subject's swing by the information processing system 1. [Figure 4] 2 is a diagram illustrating an example of a hardware configuration of an information processing terminal 10. FIG. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device 20. [Figure 6] 2 is a diagram illustrating an example of a functional configuration of an information processing terminal 10 and an information processing device 20. FIG. [Figure 7] 1 is a diagram illustrating an example of a plurality of body parts of a subject U detected from a captured moving image in an information processing system 1. FIG. [Figure 8] FIG. 10 is a diagram for explaining a score S1. [Figure 9] FIG. 10 is a diagram for explaining a score S2. [Figure 10] FIG. 10 is a diagram for explaining a score S3. [Figure 11] FIG. 10 is a diagram for explaining a score S4. [Figure 12] FIG. 10 is a diagram for explaining a score S5. [Figure 13] FIG. 10 is a diagram for explaining the score S6. [Figure 14] FIG. 10 is a diagram for explaining the score S7. [Figure 15] FIG. 10 is a diagram for explaining the score S8. [Figure 16] FIG. 10 is a diagram for explaining scores S9 to S11. [Figure 17] 10 is a sequence diagram showing an example of a processing flow in which the information processing system 1 displays an entire moving image of a subject's swing and a partial moving image of a subject's swing. [Figure 18] FIG. 10 is a sequence diagram showing another example of the processing flow in which the information processing system 1 displays an entire moving image of a subject's swing and a partial moving image of a subject's swing. [Figure 19]FIG. 10 is a sequence diagram showing yet another example of the processing flow in which the information processing system 1 displays an entire subject person's swing moving image and a partial subject person's swing moving image. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment> Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] <Outline of the information processing system> First, an overview of an information processing system according to an embodiment will be described.

[0012] An information processing system according to an embodiment calculates, based on a video of a subject performing a series of movements, a first importance obtained by multiplying a first score, which evaluates whether a posture of a first part of the subject in a first reference posture serving as a reference for the series of movements is good or bad compared to a posture of the first part of a model person in the first reference posture, by a first weight associated with the first part in the first reference posture; and a second importance obtained by multiplying a second score, which evaluates whether a posture of a second part of the subject in a second reference posture serving as a reference for the series of movements is good or bad compared to a posture of the second part of the model person in the second reference posture, by a second weight associated with the second part in the second reference posture. Here, the first weight indicates the importance of improving the posture of the first part in the first reference posture, and the second weight indicates the importance of improving the posture of the second part in the second reference posture. This allows the information processing system to identify whether improving the posture of the first part of the subject in the first reference posture or the posture of the second part of the subject in the second reference posture will be more effective in improving the series of movements of the subject. As a result, the information processing system allows the user to efficiently improve the series of movements.

[0013] The configuration of the information processing system according to the embodiment and the processing performed by the information processing system will be described in detail below.

[0014] <Configuration of information processing system> The configuration of the information processing system according to the embodiment will be described below using the information processing system 1 as an example.

[0015] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system 1. As shown in FIG.

[0016] The information processing system 1 assists the subject U in performing a series of movements based on video images of the subject U performing a predetermined series of movements. More specifically, the information processing system 1 assists the subject U so that the series of movements performed by the subject U resembles the series of movements performed by a model person. Here, the series of movements is a series of movements that a person performing a predetermined type of exercise repeatedly performs in the exercise, and is a series of movements that is composed of multiple movements that the person performs consecutively. Therefore, the series of movements includes multiple predetermined reference reference postures. The following describes, as an example, a case where the exercise is golf. In this case, the series of movements is a golf swing. A golf swing includes five predetermined reference postures: address, halfway back, top, impact, and follow-through, as shown in FIG. 2. FIG. 2 is a diagram showing the five reference postures in a golf swing arranged in chronological order. The reference posture represented by A in FIG. 2 is the address. The reference posture represented by H in FIG. 2 is the halfway back. It should be noted that halfway back is sometimes referred to as half top. The reference position represented by T in FIG. 2 is the top. The reference position represented by I in FIG. 2 is the impact. The reference position represented by F in FIG. 2 is the follow-through. For ease of explanation, these five reference positions will be simply referred to as the reference positions unless there is a need to distinguish between them. For ease of explanation, a golf swing will be simply referred to as the swing. It should be noted that the subject U may be any person. Therefore, the subject U may be a user of the information processing system 1, or may not be a user of the information processing system 1.

[0017] When the subject U makes a swing, the subject U can make the subject U's swing closer to that of a model person by bringing the subject U's reference posture closer to that of the model person. This is useful for the subject U to improve his / her golf skills. Therefore, the information processing system 1 assists the subject U so that the swing made by the subject U approaches that of the model person. The model person may be, for example, a professional golfer, a golf coach, etc., but is not limited to these. In the following, for convenience of explanation, the model person will be referred to as a target person. In other words, the information processing system 1 assists the subject U so that the swing made by the subject U approaches that of the target person.

[0018] Here, each reference posture is associated with one or more body parts that contribute to the quality of the reference posture. Each of the one or more body parts is any of the body parts of the person performing the swing. Each of the one or more body parts may be any of the body parts of the person including multiple joints of the person, or may be any of the individual joints of the person. For example, the address is associated with the buttocks and the knees of the person. Also, for example, the halfway back is associated with the buttocks of the person. Also, for example, the top is associated with the elbows and the knees of the person. Also, for example, the impact is associated with the buttocks of the person, the elbows of the person, and the center of the head of the person. Also, for example, the follow-through is associated with the center of the head of the person, the chest of the person, and the buttocks of the person. Note that the body parts associated with the address, halfway back, top, impact, and follow-through may be other body parts. In the following, for the sake of convenience, unless there is a need to distinguish between one or more parts related to the quality of each reference posture, they will be collectively referred to as reference posture related parts.

[0019] The assistance provided to the subject U by the information processing system 1 is to display a subject swing video, which is a three-dimensional video showing the swing by the subject U, and a target swing video, which is a three-dimensional video showing the swing by the target person. However, the subject swing video is a video generated by the information processing system 1, and as long as it is a video showing the swing by the subject U, it may be a video using an avatar, a video showing the trajectory of the movement of a joint point, or another type of video. As long as it is a video showing the swing by the target person, the target swing video may be a video using an avatar, a video showing the trajectory of the movement of a joint point, or another type of video. However, the target swing video may be a video generated by the information processing system 1 as needed, or may be a video stored in advance in the information processing system 1.

[0020] In displaying such a moving image, the information processing system 1 calculates, for each reference posture-related part, a score that evaluates the quality of the posture of the reference posture-related part of the subject U relative to the posture of the reference posture-related part of the target person. Details of the score that evaluates the quality of the posture of the reference posture-related part of the subject U relative to the posture of the reference posture-related part of the target person will be described later. For ease of explanation, the score that evaluates the quality of the posture of a certain reference posture-related part of the subject U relative to the posture of the reference posture-related part of the target person will be referred to as the reference posture-related part score. Here, a weight is associated with each reference posture-related part. The weight associated with a certain reference posture-related part indicates the importance of improving the posture of the reference posture-related part with respect to improving the swing. Details of the weights will be described later. After calculating each reference posture-related part score of the subject U, the information processing system 1 calculates, for each reference posture-related part of the subject U, a value obtained by multiplying the reference posture-related part score by the weight associated with the reference posture-related part as the importance of improving the posture of the reference posture-related part. After calculating the importance of each reference posture-related part of the subject U, the information processing system 1 identifies the reference posture-related part with the highest importance. For ease of explanation, the reference posture-related part identified by the information processing system 1 in this manner will be referred to as a target reference posture-related part in the following explanation.

[0021] When displaying subject swing video images, which are three-dimensional video images, the information processing system 1 displays subject swing video images in which the movements of the entire body of the virtual subject U can be seen, and subject swing video images in which the movements of the target reference posture-related parts can be seen. This allows the subject U to visually recognize the entire body of the subject U in the reference posture associated with the target reference posture-related parts, and the posture of the target reference posture-related parts. As a result, the information processing system 1 allows the subject U to efficiently improve his / her swing. As described above, the information processing system 1 also displays a target person swing video image along with these subject swing video images. This allows the subject U to visually recognize how to improve the posture of the target reference posture-related parts. Hereinafter, for convenience of explanation, the subject swing video images in which the movements of the entire body of the virtual subject U can be seen will be referred to as the subject swing entire video image, and the subject swing video images in which the movements of the target reference posture-related parts can be seen will be referred to as the subject swing partial video image.

[0022] Fig. 3 is a diagram showing an example of displaying a moving image of an entire subject's swing, a moving image of a partial subject's swing, and a moving image of a target's swing by the information processing system 1. In the example shown in Fig. 3, the information processing system 1 displays these three images on a smartphone display. In addition, in this example, the display area in which these three images are displayed is divided into two areas: a first area W1 and a second area W2.

[0023] The first area W1 is an area where the subject swing partial video image is displayed. The 3D model V1 displayed in the first area W1 is a 3D model that represents the swing of a virtual subject U in the subject swing partial video image. In the example shown in FIG. 3, the lower half of the 3D model V1 is displayed in the first area W1. This indicates that the target reference posture-related part is the lower half of the subject U. Also, in this example, a 3D model V2 is displayed superimposed on the 3D model V1. The 3D model V2 is a 3D model that represents the swing of a virtual target subject. By displaying the 3D model V1 and the 3D model V2 superimposed, the subject U can visually recognize how to change the posture of the target reference posture-related part of the subject U to approximate the posture of the target subject's target reference posture-related part.

[0024] The second area W2 is an area where the entire moving image of the subject's swing is displayed. The 3D model V3 displayed in the second area W2 is a 3D model that represents the swing of a virtual subject U in the entire moving image of the subject's swing. As shown in FIG. 3, the entire moving image of the subject's swing allows the movement of the subject U's entire body to be seen. In addition, in this example, a 3D model V4 is displayed superimposed on the 3D model V3. The 3D model V4 is a 3D model that represents the swing of a virtual target player. By displaying the 3D model V3 and the 3D model V4 superimposed, the subject U can recognize the difference between the swing of the subject U and the swing of the target player from a bird's-eye view.

[0025] 3 is merely an example. Therefore, the information processing system 1 may display the partial subject swing video, the entire subject swing video, and the target person swing video in other display modes. For example, the information processing system 1 may display the partial subject swing video, the entire subject swing video, and the target person swing video separately without overlapping each other.

[0026] Furthermore, each of the 3D models V1 to V4 may be an avatar, a human-shaped model represented by points indicating joints and links connecting the joints, or another type of 3D model.

[0027] Here, the information processing system 1 acquires moving images of the subject U swinging in order to generate moving images of a portion of the subject's swing and a moving image of the entire subject's swing. The acquisition of the moving images may be performed by the information processing system 1 itself capturing the moving images, or may be performed by acquiring the moving images captured by another device or the like from the other device. For ease of explanation, the moving images will be referred to as captured moving images below.

[0028] Furthermore, the information processing system 1 calculates the three-dimensional position of each of a plurality of predetermined body parts of the subject U at each time based on the captured video, and generates information indicating the calculated three-dimensional positions as motion data indicating the motion of the subject U. The aforementioned reference posture-related body part is any one of the predetermined body parts or a combination of two or more of the predetermined body parts. Below, as an example, a case where the reference posture-related body part is any one of the predetermined body parts will be described. Each of the predetermined body parts will be described later. The information processing system 1 generates a subject swing partial motion image and a subject swing entire motion image based on the generated motion data. The information processing system 1 also calculates each of the aforementioned reference posture-related body part scores based on the generated motion data.

[0029] The information processing system 1 includes, for example, an information processing terminal 10 and an information processing device 20. Note that the information processing system 1 may include other members, other devices, etc. in addition to the information processing terminal 10 and the information processing device 20.

[0030] In the information processing system 1, the information processing terminal 10 is connected to the information processing device 20 wirelessly so as to be able to communicate with them. Note that in the information processing system 1, the information processing terminal 10 may be connected to the information processing device 20 via a wire so as to be able to communicate with them.

[0031] The information processing terminal 10 is, for example, a multi-function mobile phone terminal (smartphone), but is not limited to this and may be other information processing terminals such as a mobile phone terminal, a tablet PC (Personal Computer), a PDA (Personal Digital Assistant), or a notebook PC.

[0032] The information processing terminal 10 includes an imaging unit C that can capture moving images. Note that the information processing terminal 10 may not include the imaging unit C, and may instead be configured to be connected to the imaging unit C as an external device.

[0033] The imaging unit C is a camera equipped with an imaging element such as a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like that converts collected light into an electrical signal. In this example, the imaging unit C is built into the information processing terminal 10. Therefore, the imaging unit C moves in accordance with the movement of the information processing terminal 10. In other words, the range that the imaging unit C can capture changes in accordance with the movement of the information processing terminal 10. The imaging unit C captures two-dimensional moving images of that range. Note that the imaging unit C may also be configured to be capable of capturing still images of that range.

[0034] In response to the received operation, the information processing terminal 10 causes the imaging unit C to capture a moving image within the range that the imaging unit C can capture. This allows the information processing terminal 10 to capture a moving image of the subject U performing a swing motion as the captured moving image described above. After capturing the captured moving image, the information processing terminal 10 transmits moving image data of the captured captured moving image to the information processing device 20.

[0035] After transmitting the moving image data of the captured moving image to the information processing device 20, the information processing terminal 10 waits until it receives, as a response from the information processing device 20, moving image data of the entire subject's swing moving image and target reference posture-related body part information indicating the target reference posture-related body part. When the information processing terminal 10 receives the moving image data and the target reference posture-related body part information, it generates partial subject's swing moving images based on the received moving image data and target reference posture-related body part information. The information processing terminal 10 then displays the entire subject's swing moving image of the moving image data and the generated partial subject's swing moving images. At this time, the information processing terminal 10 displays the target subject's swing moving image together with the entire subject's swing moving image and the partial subject's swing moving images in accordance with the received operation. Note that the information processing terminal 10 may be configured to receive the target subject's swing moving image from the information processing device 20, may be configured to store a predetermined target subject's swing moving image, or may be configured to generate the target subject's swing moving image.

[0036] The information processing device 20 may be any information processing device that can function as a server. For example, the information processing device 20 may be a workstation, a desktop PC, a notebook PC, or the like, but is not limited to these.

[0037] The information processing device 20 waits until it receives moving image data of the captured moving image from the information processing terminal 10. When the information processing device 20 receives the moving image data, it generates an entire moving image of the subject's swing based on the captured moving images of the received moving image data. The information processing device 20 also calculates each reference posture-related part score based on the captured moving images, and identifies a target reference posture-related part based on each calculated reference posture-related part score. The information processing device 20 transmits target reference posture-related part information indicating the identified target reference posture-related part and the generated moving image of the entire swing of the subject to the information processing terminal 10.

[0038] <Hardware configuration of information processing terminal> The hardware configuration of the information processing terminal 10 will be described below with reference to Fig. 4. Fig. 4 is a diagram showing an example of the hardware configuration of the information processing terminal 10.

[0039] The information processing terminal 10 includes, for example, a first processor 11, a first storage unit 12, an input receiving unit 13, a first communication unit 14, a display unit 15, and an imaging unit C. These components are connected to each other via a bus so that they can communicate with each other. The information processing terminal 10 also communicates with the information processing device 20 via the first communication unit 14.

[0040] The first processor 11 is, for example, a CPU (Central Processing Unit). Note that the first processor 11 may be another processor such as an FPGA (Field Programmable Gate Array) instead of a CPU. The first processor 11 executes various programs stored in the first storage unit 12.

[0041] The first storage unit 12 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), etc. Note that the first storage unit 12 may be an external storage device connected via a digital input / output port such as a universal serial bus (USB) instead of being built into the information processing terminal 10. The first storage unit 12 stores various types of information, images, programs, etc. that are processed by the information processing terminal 10.

[0042] The input receiving unit 13 is an input device, and is, for example, a touch panel that is configured integrally with the display unit 15. Note that the input receiving unit 13 may be another input device, such as a keyboard, a mouse, or a touchpad, that is configured separately from the display unit 15, instead of the touch panel.

[0043] The first communication unit 14 includes, for example, a digital input / output port such as a USB, an Ethernet (registered trademark) port, an antenna for wireless communication, and the like.

[0044] The display unit 15 is a display device, and includes, for example, a liquid crystal display.

[0045] <Hardware configuration of information processing device> The hardware configuration of the information processing device 20 will be described below with reference to Fig. 5. Fig. 5 is a diagram showing an example of the hardware configuration of the information processing device 20.

[0046] The information processing device 20 includes, for example, a second processor 21, a second storage unit 22, and a second communication unit 24. These components are connected to each other via a bus so that they can communicate with each other. The information processing device 20 also communicates with the information processing terminal 10 via the second communication unit 24.

[0047] The second processor 21 is, for example, a CPU. Note that the second processor 21 may be another processor such as an FPGA instead of a CPU. The second processor 21 executes various programs stored in the second storage unit 22.

[0048] The second storage unit 22 includes, for example, an HDD, an SSD, an EEPROM, a ROM, a RAM, etc. Note that the second storage unit 22 may be an external storage device connected via a digital input / output port such as a USB, instead of being built into the information processing device 20. The second storage unit 22 stores various types of information, various images, various programs, etc. to be processed by the information processing device 20.

[0049] The second communication unit 24 includes, for example, a digital input / output port such as a USB, an Ethernet (registered trademark) port, an antenna for wireless communication, and the like.

[0050] <Functional configuration of information processing terminal and information processing device> The functional configurations of the information processing terminal 10 and the information processing device 20 will be described below with reference to Fig. 6. Fig. 6 is a diagram showing an example of the functional configurations of the information processing terminal 10 and the information processing device 20.

[0051] The information processing terminal 10 includes a first storage unit 12, an input receiving unit 13, a first communication unit 14, a display unit 15, an imaging unit C, and a first control unit 16.

[0052] The first control unit 16 controls the entire information processing terminal 10. The first control unit 16 includes an imaging control unit 161, a first processing unit 162, and a display control unit 163. These functional units included in the first control unit 16 are realized, for example, by the first processor 11 executing various programs stored in the first storage unit 12. Some or all of the functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

[0053] The imaging control unit 161 controls the imaging unit C.

[0054] The first processing unit 162 performs various processes performed by the information processing terminal 10.

[0055] The display control unit 163 generates various images in response to operations received via the input receiving unit 13. The display control unit 163 displays the generated images on the display unit 15. Furthermore, the display control unit 163 displays moving images of moving image data received from the information processing device 20 on the display unit 15 in response to operations received via the input receiving unit 13.

[0056] The information processing device 20 includes a second storage unit 22, a second communication unit 24, and a second control unit .

[0057] The second control unit 26 controls the entire information processing device 20. The second control unit 26 includes a second processing unit 261. The second processing unit 261 included in the second control unit 26 is realized, for example, by the second processor 21 executing various programs stored in the second storage unit 22. The second processing unit 261 may also be a hardware functional unit such as an LSI or an ASIC.

[0058] The second processing unit 261 performs various processes performed by the information processing device 20.

[0059] <Multiple parts of a subject detected from captured video images> Hereinafter, with reference to FIG. 7, multiple body parts of the subject U detected from captured moving images in the information processing system 1 will be described. FIG. 7 is a diagram illustrating multiple body parts of the subject U detected from captured moving images in the information processing system 1. In the example shown in FIG. 7, each of the multiple body parts of the subject U is represented by a node connected by a straight line called a link. Node 0.Pelvis indicates the waist of the subject U. Node 1.R.Hip indicates the right buttock of the subject U. Node 2.R.Knee indicates the right knee of the subject U. Node 3.R.Ankle indicates the right ankle of the subject U. Node 4.L.Hip indicates the left buttock of the subject U. Node 5.L.Knee indicates the left knee of the subject U. Node 6.L.Ankle indicates the left ankle, one of the parts of subject U's body. Node 7.Spine indicates the spine, one of the parts of subject U's body. Node 8.Thorax indicates the thorax, one of the parts of subject U's body. Node 9.Neck indicates the neck, one of the parts of subject U's body. Node 10.Head indicates the center of the head, one of the parts of subject U's body. Node 11.L.Shoulder indicates the left shoulder, one of the parts of subject U's body. Node 12.L.Elbow indicates the left elbow, one of the parts of subject U's body. Node 13.L.Wrist indicates the left wrist, one of the parts of subject U's body. Node 14.R.Shoulder indicates the right shoulder, one of the parts of subject U's body. Node 15.R.Elbow indicates the right elbow, one of the parts of subject U's body. Node 16.R.Wrist indicates the right wrist of subject U's body parts.In the following, for convenience of explanation, node 0.Pelvis will be referred to as node 0, node 1.R.Hip will be referred to as node 1, node 2.R.Knee will be referred to as node 2, node 3.R.Ankle will be referred to as node 3, node 4.L.Hip will be referred to as node 4, node 5.L.Knee will be referred to as node 5, node 6.L.Ankle will be referred to as node 6, node 7.Spine will be referred to as node 7, and node 8.Thorax will be referred to as node 8. In this description, node 9.Neck will be referred to as node 9, node 10.Head will be referred to as node 10, node 11.L.Shoulder will be referred to as node 11, node 12.L.Elbow will be referred to as node 12, node 13.L.Wrist will be referred to as node 13, node 14.R.Shoulder will be referred to as node 14, node 15.R.Elbow will be referred to as node 15, and node 16.R.Wrist will be referred to as node 16.

[0060] A distance from node 0 is associated with each of the body parts of subject U represented by nodes 1 to 16. The distance between node 0 and a certain node among nodes 1 to 16 is represented by the number of links passed from that node to node 0 in the shortest route. For example, the distance associated with the spine, which is a body part of subject U represented by node 7, is 1 because the number of links passed from node 7 to node 0 in the shortest route is 1. Furthermore, the distance associated with the left shoulder, which is a body part of subject U represented by node 11, is 3 because the number of links passed from node 11 to node 0 in the shortest route is 3. Note that 0 is associated with the waist, which is a body part of subject U represented by node 0, as this distance.

[0061] <Examples of reference posture-related body part scores> A specific example of the reference posture related part scores will be described below. As an example, the reference posture related part scores will be described below as eleven scores S1 to S11.

[0062] FIG. 8 is a diagram for explaining the score S1. The score S1 is a score for the buttocks associated with the address. The score S1 is calculated based on the Δθ A1 and Δθ A2 The sum of 2 The value is calculated by the index value IX1 divided by the following formula (1). Score S1 = 100 - (index value IX1) × (60 / 100) (1)

[0063] In the example shown in Fig. 8, the posture of the subject U is at address. The straight line shown in Fig. 8 is an arrow indicating a direction tilted 60° from the horizontal line toward the front and above of the subject U shown in Fig. 8.

[0064] Δθ A1 is 60° to θ A1 It is defined as the value obtained by subtracting θ A1 is the angle between the horizontal line and the line connecting the right shoulder of the subject U and the center of the head of the subject U. A1 is the target's θ A1 is approximately 60°, this is an amount that indicates the degree to which the tilt of the subject U's head relative to his / her right shoulder is different from the tilt of the target's head relative to his / her right shoulder.

[0065] On the other hand, Δθ A2 is 60° to θ A2 It is defined as the value obtained by subtracting θ A2 is the narrow angle between the horizontal line and the line connecting the subject's right buttock and the waist of subject U. A2 is the target's θ A2 is approximately 60°, this is an amount that indicates the degree to which the inclination of the subject U's waist relative to his right buttocks is different from the inclination of the target's waist relative to his right buttocks.

[0066] And R 2 is the coefficient of determination in the regression of the ground (i.e., the horizon), subject U's right buttock, subject U's waist, and subject U's right shoulder.

[0067] Therefore, the smaller the index value IX1, the closer the address of the subject U is to the address of the target. In other words, the larger the score S1, the closer the address of the subject U is to the address of the target.

[0068] FIG. 9 is a diagram for explaining the score S2. The score S2 is a score for the knee associated with the address. The score S2 is calculated by dividing the score by the X RW and W RA and an index value IX2, which is the absolute value of the difference between the values, and the value calculated by the following formula (2). Score S2 = 100 - (index value IX2) × 1000 (2)

[0069] In the example shown in FIG. 9, the posture of the subject U is at address. RW indicates the position of the right buttock of the subject U in the horizontal direction, that is, the X coordinate of the right buttock of the subject U. Also, the X RA indicates the position of the right ankle of the subject U in the horizontal direction, that is, the X coordinate of the right ankle of the subject U. Note that the index value IX2 is RW and X RA The origin of the X coordinate may be at any position because the index value IX2 is the absolute value of the difference between the right buttock and the right ankle of the subject U. In other words, the index value IX2 indicates the horizontal distance between the right buttock and the right ankle of the subject U.

[0070] When the target person is in address position, the horizontal distance between the target person's right buttock and right ankle is approximately 0. Therefore, the smaller the index value IX2, the closer the address of the target person U is to the address of the target person. In other words, the larger the score S2, the closer the address of the target person U is to the address of the target person.

[0071] FIG. 10 is a diagram for explaining the score S3. The score S3 is a score for the waist associated with the halfway back. The score S3 is calculated based on the θ B1 , θ B2 , θB3 and θ when the posture of the subject U is halfway back H1 , θ H2 , θ H3 and an index value IX3 calculated by the following formula (3): Score S3 = 100 - (index value IX3) × (100 / 45) (3)

[0072] The subject U on the left side of FIG. 10 is the subject U in the address posture. In the example shown in FIG. B1 is the narrow angle between the horizontal line and the straight line connecting the right buttock and the center of the head of the target U in the address posture. B2 is the angle between the horizontal line and the line connecting the right ankle and the center of the head of the target U in the address position. B3 is the flexion angle of the right knee of the subject U in the address posture. Note that in this embodiment, the flexion angle of the right knee is the widest angle among the angles at which the extension direction of the right thigh and the extension direction of the right lower leg intersect.

[0073] On the other hand, the subject U on the right side of Fig. 10 is a subject U with a posture of halfway back. H1 is the narrow angle between the horizontal line and the line connecting the right buttock and the center of the head of the subject U who is in a halfway back posture. H2 is the narrow angle between the horizontal line and the line connecting the right ankle and the center of the head of the subject U who is in a halfway back posture. H3 is the angle of flexion of the right knee of subject U in a halfway back position.

[0074] Here, the index value IX3 is a value defined as the sum of Δθ1, Δθ2, and Δθ3. H1 From θ B1 Also, Δθ2 is the value obtained by subtracting θ H2 From θ B2 Also, Δθ3 is the value obtained by subtracting θ H3 From θ B3 The value is the difference between the two.

[0075] The tilt of the head relative to the right buttocks at the target's halfway back is almost the same as the tilt of the head relative to the right buttocks at the target's address. This means that the target's Δθ1 is almost 0. Furthermore, the tilt of the head relative to the right ankle at the target's halfway back is almost the same as the tilt of the head relative to the right ankle at the target's address. This means that the target's Δθ2 is almost 0. Furthermore, the flexion angle of the right knee at the target's halfway back is almost the same as the flexion angle of the right knee at the target's address. This means that the target's Δθ3 is almost 0. Therefore, the smaller the index value IX3, the closer the halfway back of the subject U is to the halfway back of the target. In other words, the larger the score S3, the closer the halfway back of the subject U is to the halfway back of the target.

[0076] 11 is a diagram illustrating the score S4, which is the score for the elbow associated with the top player.

[0077] In the example shown in Fig. 11, the posture of the subject U is at the top of the head. The angle θ shown in Fig. 11 is the angle between the horizontal line and a line connecting the right elbow and right wrist of the subject U shown in Fig. 11.

[0078] The score S4 is a value calculated using the index value IX4 defined as the value obtained by subtracting θ from 90° and the following formula (4). Score S4 = 100 - (index value IX4) × (100 / 90) (4) Here, the target's θ is approximately 90°. Therefore, the smaller the index value IX4, the closer the top of subject U is to the top of the target. In other words, the larger the score S4, the closer the top of subject U is to the top of the target.

[0079] FIG. 12 is a diagram illustrating the score S5. The score S5 is a score for the knee associated with the top person. The score S5 is calculated based on the θ C and θ when the subject U is in the top position. T and an index value IX5 calculated by the following formula (5): Score S5 = 100 - (index value IX5) × (100 / 45) (5)

[0080] The subject U on the left side of FIG. 12 is the subject U in the address posture. In the example shown in FIG. C is the angle of flexion of the right knee of subject U in the address position.

[0081] On the other hand, the subject U on the right side of Fig. 12 is the subject U with the top posture. T is the flexion angle of the right knee of subject U when he is at the top of his posture.

[0082] Here, the index value IX5 is θ C From θ T The index value IX5 is defined as the value obtained by subtracting the flexion angle of the right knee at the top of the swing from the target's knee. The flexion angle of the right knee at the top of the swing of the target is almost the same as the flexion angle of the right knee at the target's address. Therefore, the smaller the index value IX5, the closer the top of the swing of the target U is to the top of the swing of the target. In other words, the larger the score S5, the closer the top of the swing of the target U is to the top of the swing of the target.

[0083] FIG. 13 is a diagram for explaining the score S6. Note that FIG. 13 shows an arrow indicating the X-axis direction in FIG. 13. The score S6 is a score for the buttocks associated with the impact. The score S6 is calculated by dividing the X-axis direction shown in FIG. 13 by the score S6. RS , X LS , X RW , X LW and n, which will be described later, and an index value IX6 calculated by the following formula (6). Score S6 = (index value IX6) × 100 (6)

[0084] The target U shown in FIG. 13 is the target U whose posture is impact. In the example shown in FIG. RS indicates the position in the X-axis direction of the right shoulder of the target U whose posture is impacted, that is, the X coordinate of the right shoulder of the target U whose posture is impacted. LS indicates the horizontal position of the left shoulder of the target U whose posture is impacted, that is, the X coordinate of the left shoulder of the target U whose posture is impacted. RW indicates the horizontal position of the right hip of the subject U whose posture is impact, that is, the X coordinate of the right hip of the subject U whose posture is impact. LW indicates the horizontal position of the left hip of the subject U whose posture is impact, that is, the X coordinate of the left hip of the subject U whose posture is impact.

[0085] The aforementioned n is the number of frames that satisfy the following predetermined condition among the 10 frames immediately before the impact out of all frames included in the captured video.

[0086] Given condition: X RS <X LS or X RS -X LS <0.01

[0087] The index value IX6 is calculated using ΔX′ defined as in the following equation (7).

[0088] ΔX'=(X RW -X LW ) / max_swing ΔX (7)

[0089] Here, max_swing ΔX in the above formula (7) indicates the maximum value of the distance between the right hip and the left hip of the subject U in the X-axis direction throughout one swing of the subject U in the captured video. The index value IX6 is defined as the value obtained by dividing n by 10 and multiplying it by ΔX'.

[0090] The target person's shoulders often satisfy the above-mentioned predetermined conditions immediately before impact. Furthermore, the distance between the target person's right hip and left hip in the X-axis direction is close to the maximum value when the target person is in an impact posture. Therefore, the larger the index value IX6, the closer the target person U's impact is to the target person's impact. In other words, the larger the score S6, the closer the target person U's impact is to the target person's impact.

[0091] The above specified conditions are X RS and X LS In addition, in the above formula (1), (X RW -X LW ) / max_swing ΔX is X RW and X LW Therefore, the origin of the X coordinate can be at any position.

[0092] FIG. 14 is a diagram for explaining the score S7. Note that FIG. 14 shows arrows indicating the X-axis direction and the Y-axis direction in FIG. 14. The score S7 is a score for the elbow associated with the impact. The score S7 is calculated based on the X-coordinate X of the right elbow of the subject U when the subject U is in the address position. A-RE and Y coordinate Y A-RE and the X coordinate X of the right elbow of the subject U when the subject U is in the impact position. I-RE and Y coordinate Y I-RE and an index value IX7 calculated by the following equation (8). Score S7 = (100 - (index value IX7) × 2000) / 2 (8) The subject U on the left side of Fig. 14 is the subject U in the address posture, and the subject U on the right side of Fig. 14 is the subject U in the impact posture.

[0093] Here, the index value IX7 is defined as Δd in the following equation (9).

[0094] Δd=(|(XA-RE ,Y A-RE )-(X I-RE ,Y I-RE )|)×d ···(9)

[0095] d in the above formula (9) is X A-RE ≦X I-RE If X is A-RE >X I-RE In this embodiment, |(X A-RE ,Y A-RE )-(X I-RE ,Y I-RE )| is ((X A-RE ,Y A-RE )-(X I-RE ,Y I-RE )) norm. The X and Y coordinates of the target's right elbow at impact are almost the same as the X and Y coordinates of the target's right elbow at address. Therefore, the smaller the index value IX7, the closer the impact of the target U is to the impact of the target. In other words, the larger the score S7, the closer the impact of the target U is to the impact of the target.

[0096] FIG. 15 is a diagram for explaining the score S8. Note that FIG. 15 shows an arrow indicating the Y-axis direction in FIG. 15. The score S8 is a score for the center of the head associated with the impact. The score S8 is calculated by dividing the Y coordinate Y of the center of the head of the subject U when the subject U is in the address position. A and the Y coordinate Y of the center of the head of the subject U when the subject U is in the impact position. I and a value calculated by the following formula (10): Score S8 = 100 - (index value IX8) × 20 (10) The subject U on the left side of Fig. 15 is the subject U in the address posture, and the subject U on the right side of Fig. 15 is the subject U in the impact posture.

[0097] Here, the index value IX8 is the Y coordinate Y A to Y coordinate YI The score S8 is defined as the value obtained by subtracting the Y coordinate of the center of the target's head at impact. The Y coordinate of the center of the target's head at impact is almost the same as the Y coordinate of the center of the eastern part of the target's address. Therefore, the smaller the index value IX8, the closer the impact of the target U is to the impact of the target. In other words, the larger the score S8, the closer the impact of the target U is to the impact of the target.

[0098] FIG. 16 is a diagram for explaining scores S9 to S11. Note that FIG. 16 shows an arrow indicating the Y-axis direction in FIG. 16. Score S9 is the score for the center of the head associated with the follow-through. Score S10 is the score for the chest associated with the follow-through. Score S11 is the score for the buttocks associated with the follow-through.

[0099] The score S9 is the Y coordinate Y of the center of the head of the subject U when the subject U is in the address position. A and the Y coordinate Y of the center of the head of the subject U when the subject U's posture is follow-through. F and an index value IX9 calculated by the following equation (11). Score S9 = 100 - (index value IX9) × 20 (11) The subject U on the left side of FIG. 16 is a subject U with an address posture. The subject U on the right side of FIG. 16 is a subject U with a follow-through posture. Here, the index value IX9 is expressed as the Y coordinate Y F to Y coordinate Y A The Y coordinate of the center of the target's head during the follow-through is approximately the same as the Y coordinate of the center of the eastern part of the target's face during the address. Therefore, the smaller the index value IX9, the closer the follow-through of the target U is to the follow-through of the target. In other words, the larger the score S9, the closer the follow-through of the target U is to the follow-through of the target.

[0100] The score S10 is calculated by the narrow angle θ between the horizontal line and the line connecting the thorax of the subject U and the center of the head of the subject U when the subject U is in the address position. A-SH and the narrow angle θ between the horizontal line and the line connecting the thorax of the subject U and the center of the head of the subject U when the posture of the subject U is follow-through. F-SH and a value calculated by the following formula (12): Score S10 = 100 - (index value IX10) × (100 / 45) (12) Here, the index value IX10 is F-SH From θ A-SH The index value IX10 is defined as the value obtained by subtracting the target's head tilt from the rib cage during the target's follow-through. The tilt of the target's head relative to the rib cage during the target's address is almost the same as the tilt of the target's head relative to the rib cage during the target's address. Therefore, the smaller the index value IX10, the closer the target U's follow-through is to the target's follow-through. In other words, the larger the score S10, the closer the target U's follow-through is to the target's follow-through.

[0101] The score S11 is calculated by the narrow angle θ between the horizontal line and the line connecting the left buttock of the subject U and the center of the head of the subject U when the subject U is in the address position. A-LHH and the index value IX11 calculated using the narrow angle θF-LHH between the horizontal line and the line connecting the left buttock of subject U and the center of subject U's head when subject U's posture is follow-through, and the value calculated using the following equation (13). Score S11 = 100 - (index value IX11) × 20 (13) Here, the index value IX11 is θ F-LHH From θ A-LHH The index value IX11 is defined as the value obtained by subtracting the following. The tilt of the target's head relative to the left buttocks during the follow-through is almost the same as the tilt of the target's head relative to the left buttocks during the address. Therefore, the smaller the index value IX11, the closer the follow-through of the target U is to the follow-through of the target U. In other words, the larger the score S11, the closer the follow-through of the target U is to the follow-through of the target U.

[0102] Note that the reference posture-related parts associated with each of the 11 reference posture-related part scores described in FIGS. 8 to 16 may be replaced with parts indicated by the 17 nodes shown in FIG. 7. In this case, these 17 parts are associated with each reference posture as reference posture-related parts. In this case, the reference posture-related part score is represented by the distance between the part of the subject U and the part of the target person. For example, the reference posture-related part score for the right shoulder among the reference posture-related part scores associated with an address is the distance between the right shoulder of the subject U in the address posture and the right shoulder of the target person in the address posture. In this case, the information processing system 1 pre-stores information indicating the three-dimensional position of each of the 17 parts of the target person at each time as motion data indicating the motion of the target person. Therefore, based on the calculated motion data of the subject U and the pre-stored motion data of the target person, the information processing system 1 calculates the distance between the right shoulder of the subject U in the address posture and the right shoulder of the target person in the address posture as the reference posture-related part score for the right shoulder among the reference posture-related part scores associated with the address. However, the information processing system 1 matches the three-dimensional position of the waist of the subject U with the three-dimensional position of the waist of the target person, thereby matching a three-dimensional coordinate system indicating the three-dimensional position of each of the multiple body parts of the subject U with a three-dimensional coordinate system indicating the three-dimensional position of each of the multiple body parts of the target person. Such a process for matching the three-dimensional coordinate systems may be a known process or a process to be developed in the future.

[0103] Furthermore, the 11 scores S1 to S11 must be comparable to each other. For this reason, these 11 scores are adjusted to comparable values ​​using the above formulas (1) to (6), (8), and (10) to (13). This is a result obtained empirically through experiments. For this reason, each of the above formulas (1) to (6), (8), and (10) to (13) may be replaced by another formula. Furthermore, each of the above formulas (1) to (6), (8), and (10) to (13) may be a formula derived from some theoretical formula.

[0104] <Weights associated with reference posture-related parts> The weights associated with the reference posture related parts will be described below. The weights associated with the reference posture related parts are determined based on predetermined conditions. The predetermined conditions include, for example, the following two conditions, Condition 1 and Condition 2.

[0105] Condition 1: The earlier the reference posture is executed in the swing, the greater the weight is assigned to it. Condition 2: The shorter the distance associated with the reference posture-related part, the larger the weight associated with it.

[0106] Condition 1 is adopted because it is believed that the closer a reference posture is to the address, the more negatively it affects the reference posture in the latter half of the swing. For example, based on condition 1, the weights associated with parts related to the reference posture at address are greater than the weights associated with parts related to the reference posture at halfway back. Condition 2 is adopted because it is believed that the closer a part is to the waist, the more negatively it affects the quality of the reference posture in a swing. For example, based on condition 2, the weight associated with the right shoulder is greater than the weight associated with the right wrist. Note that the predetermined condition may include either condition 1 or condition 2. For example, condition 1 may be replaced by a condition in which a greater weight is associated with a reference posture that is closer to impact when executed in a swing. For example, condition 1 may be replaced by a condition in which a greater weight is associated with a reference posture that is closer to the reference posture that is easier for a person to correct when executed in a swing. Note that condition 1 may be replaced by another condition. Note that condition 2 may be replaced by another condition.

[0107] <Process for displaying moving images of the entire subject's swing and moving images of parts of the subject's swing> Hereinafter, with reference to FIG. 17 , a process in which the information processing system 1 displays a subject person's entire swing moving image and a subject person's partial swing moving image will be described. FIG. 17 is a sequence diagram showing an example of a process flow in which the information processing system 1 displays a subject person's entire swing moving image and a subject person's partial swing moving image. Hereinafter, as an example, a case will be described in which, at a timing before the process of step S110 shown in FIG. 17 is performed, the information processing terminal 10 captures a captured moving image and the captured captured moving image is stored in the first storage unit 12 of the information processing terminal 10. Hereinafter, as an example, a case will be described in which, at that timing, the information processing terminal 10 receives a process start operation for starting a process for displaying a subject person's entire swing moving image and a subject person's partial swing moving image. Hereinafter, as an example, a case will be described in which, at that timing, first correspondence information that associates a reference posture with a reference posture-related part associated with the reference posture is stored in the second storage unit 22 of the information processing device 20. The first correspondence information is, for example, information in a table format, but is not limited to this. In the following, as an example, a case will be described in which second correspondence information associating reference posture-related parts with weights associated with the reference posture-related parts is stored in the second storage unit 22 of the information processing device 20. The second correspondence information is, for example, information in a table format, but is not limited to this. Note that the first correspondence information and the second correspondence information may be combined into one piece of information. In the following, as an example, a case will be described in which third correspondence information associating a 3D video displayed on the information processing terminal 10 with an initial viewpoint of the 3D video is pre-stored in the first storage unit 12. The third correspondence information is information in a table format, but is not limited to this. In the following, as an example, a case will be described in which the user of the information processing terminal 10 is the subject U himself / herself.

[0108] After receiving the processing start operation, the first processing unit 162 transmits the moving image data of the captured moving image stored in the first storage unit 12 and user information related to the user of the information processing terminal 10 to the information processing device 20 via the first communication unit 14 (step S110). Here, the user information includes user identification information for identifying the user, a password unique to the user, terminal identification information for identifying the information processing terminal 10, etc., but is not limited to these.

[0109] After the information processing terminal 10 transmits the video data and the user information in step S110, the second processing unit 261 receives the video data and the user information transmitted from the information processing terminal 10 via the second communication unit 24 (step S210). In step S210, the second processing unit 261 determines whether the received user information is stored in the second storage unit 22. If the second processing unit 261 determines that the received user information is not stored in the second storage unit 22, the second processing unit 261 associates the received user information with the received video data and stores them in the second storage unit 22. On the other hand, if the second processing unit 261 determines that the received user information is stored in the second storage unit 22, the second processing unit 261 associates the received user information with the user information stored in the second storage unit 22 and stores the received video data in the second storage unit 22.

[0110] Next, the second processing unit 261 calculates exercise data indicating the exercise of the subject U based on the captured moving images of the moving image data received in step S210 (step S220). The method for calculating the exercise data based on the captured moving images may be a known method or a method to be developed in the future. After calculating the exercise data, the second processing unit 261 stores the calculated exercise data in the second storage unit 22 in association with the user information stored in the second storage unit 22.

[0111] Next, the second processing unit 261 generates a video of the entire subject's swing based on the motion data calculated in step S220 (step S230). In step S230, the second processing unit 261 generates a video of the entire subject's swing based on, for example, inverse kinematics that is constrained to resemble human movement. The method for generating the video of the entire subject's swing based on the motion data may be a known method or a method to be developed in the future. After generating the video of the entire subject's swing, the second processing unit 261 stores the generated video of the entire subject's swing in the second storage unit 22 in association with the user information stored in the second storage unit 22.

[0112] Next, the second processing unit 261 reads out the first correspondence information and the second correspondence information stored in advance in the second storage unit 22 from the second storage unit 22. Then, the second processing unit 261 calculates the importance of each reference posture-related part of the subject U based on the read first correspondence information and second correspondence information and the motion data calculated in step S220 (step S240). The method of calculating the importance has already been explained, so a repeated explanation will be omitted here. After calculating the importance, the second processing unit 261 stores the calculated importance in the second storage unit 22 in association with the user information stored in the second storage unit 22.

[0113] Next, the second processing unit 261 identifies the target reference posture-related part based on the importance calculated in step S240 (step S250). After identifying the target reference posture-related part, the second processing unit 261 stores, in the second storage unit 22, target reference posture-related part information indicating the identified target reference posture-related part in association with the user information stored in the second storage unit 22.

[0114] Next, the second processing unit 261 transmits the target reference posture related part information indicating the target reference posture related part identified in step S250, together with the video data of the subject's entire swing video image generated in step S230, to the information processing terminal 10 via the second communication unit 24 (step S260).

[0115] After the information processing device 20 transmits the moving image data and the target reference posture related body part information in step S260, the first processing unit 162 receives the moving image data and the target reference posture related body part information transmitted from the information processing device 20 via the first communication unit 14 (step S120).

[0116] Next, the first processing unit 162 generates a partial moving image of the subject's swing based on the entire moving image of the subject's swing of the moving image data received in step S120 and the target reference posture related part indicated by the target reference posture related part information received in step S120 (step S130).

[0117] Next, the display control unit 163 reads out third correspondence information pre-stored in the first storage unit 12 from the first storage unit 12. Then, the display control unit 163 causes the display unit 15 to display, using an initial viewpoint based on the read third correspondence information, each of the subject person's swing partial video generated by the first processing unit 162 in step S130 and the subject person's entire swing video of the video data received in step S120 (step S140). Note that the initial viewpoint of the three-dimensional video may be different for each three-dimensional video, or may be a common viewpoint for some or all of the three-dimensional videos. Furthermore, in step S140, the display control unit 163 may be configured to display, on the display unit 15, an estimated swing video showing a swing that the subject person U is estimated to perform if the posture of the target reference posture-related part of the subject person U is improved, together with the subject person's swing partial video and the subject person's entire swing video, in response to a previously accepted operation. In this case, in step S140, the first processing unit 162 generates an estimated swing video showing a swing that the subject U is estimated to perform when the posture of the target reference posture-related part of the subject U is brought closer to the posture of the target reference posture-related part of the target person, based on the subject person's swing partial video and the subject person's swing entire video, and on inverse kinematics. The method for generating the estimated swing video may be a known method or a method to be developed in the future. This enables the information processing system 1 to enable the user to improve their swing more reliably and efficiently. Note that the display control unit 163 may be configured to display the estimated swing video on the display unit 15 at any other timing after step S140.

[0118] Through the above-described processing, the information processing system 1 displays a moving image of the entire subject's swing and a moving image of a portion of the subject's swing, thereby enabling the user to efficiently improve their swing.

[0119] The captured video may be a video showing each of a plurality of swings of the subject U. In this case, in step S240, the second processing unit 261 calculates the importance of each reference posture-related part of the subject U for each of the plurality of swings. Then, the second processing unit 261 calculates the average value of the importance for each reference posture-related part. After calculating the average value of the importance for each reference posture-related part, in step S250, the second processing unit 261 identifies the target reference posture-related part using the calculated average value. This allows the information processing system 1 to identify the target reference posture-related part with higher accuracy.

[0120] <First Modification of the Embodiment> Hereinafter, a first modification of the embodiment will be described with reference to FIG. 18. In the first modification of the embodiment, the information processing terminal 10 generates each of the entire subject person's swing moving image and the partial subject person's swing moving image. FIG. 18 is a sequence diagram showing another example of a processing flow in which the information processing system 1 displays the entire subject person's swing moving image and the partial subject person's swing moving image. Note that the processes of steps S110, S130, and S140 shown in FIG. 18 are similar to the processes of steps S110, S130, and S140 shown in FIG. 17. Therefore, repeated description of the processes of steps S110, S130, and S140 shown in FIG. 18 will be omitted. Furthermore, the processes of steps S210, S220, S240, and S250 shown in FIG. 18 are similar to the processes of steps S210, S220, S240, and S250 shown in FIG. 17. Therefore, repeated description of the processes of step S210, step S220, step S240, and step S250 shown in Fig. 18 will be omitted. Also, as shown in Fig. 18, in Modification 1 of the embodiment, information processing device 20 does not perform processing equivalent to the processing of step S230 shown in Fig. 17. That is, in Modification 1 of the embodiment, information processing device 20 does not generate a moving image of the entire subject person's swing.

[0121] After the processing of step S250 is performed, the second processing unit 261 transmits the target reference posture related part information indicating the target reference posture related part identified in step S250, together with the motion data calculated in step S220, to the information processing terminal 10 via the second communication unit 24 (step S410).

[0122] After the information processing device 20 transmits the exercise data and the target reference posture related part information in step S270, the first processing unit 162 receives the exercise data and the target reference posture related part information transmitted from the information processing device 20 via the first communication unit 14 (step S310).

[0123] Next, the first processing unit 162 generates a moving image of the entire swing of the subject based on the motion data received in step S310 (step S320). The process of step S320 is the same as the process of step S230 shown in Fig. 17. Therefore, a repeated description of the process of step S320 will be omitted here.

[0124] Next, the first processing unit 162 generates a partial subject swing moving image based on the subject entire swing moving image generated in step S320 and the subject reference posture related part indicated by the subject reference posture related part information received in step S310 (step S330).

[0125] Then, in step S140, the display control unit 163 reads out the third correspondence information pre-stored in the first memory unit 12 from the first memory unit 12, and displays on the display unit 15, using an initial viewpoint based on the read third correspondence information, the partial motion image of the subject's swing generated by the first processing unit 162 in step S330 and the entire motion image of the subject's swing generated by the first processing unit 162 in step S320.

[0126] Through the above-described processing, the information processing system 1 also displays a moving image of the entire subject's swing and a moving image of a portion of the subject's swing. This also allows the information processing system 1 to allow the user to efficiently improve their swing.

[0127] <Modification 2 of the embodiment> Hereinafter, a second modification of the embodiment will be described with reference to FIG. 19 . In the second modification of the embodiment, the information processing terminal 10 generates the entire subject person swing moving image, the partial subject person swing moving image, the exercise data, and the target reference posture-related body part information. FIG. 19 is a sequence diagram showing yet another example of the processing flow in which the information processing system 1 displays the entire subject person swing moving image and the partial subject person swing moving image. Note that the processing of steps S110 and S140 shown in FIG. 19 is similar to the processing of step S110 shown in FIG. 17 . Therefore, a repeated description of the processing of steps S110 and S140 shown in FIG. 19 will be omitted. Also, the processing of step S210 shown in FIG. 19 is similar to the processing of step S210 shown in FIG. 17 . Therefore, a repeated description of the processing of step S210 shown in FIG. 19 will be omitted. Also, as shown in FIG. 19 , in the second modification of the embodiment, the information processing device 20 does not perform processing equivalent to the processing of steps S220 to S260 shown in FIG. 17 . In the following, as an example, a case will be described in which the first to third correspondence information are each stored in advance in the first storage unit 12 at a timing before the processing of step S110 shown in FIG. 19 is performed.

[0128] After the process of step S110 is performed, the first processing unit 162 calculates motion data indicating the motion of the subject U based on the captured moving images stored in the first storage unit 12 (step S510). The process of step S510 is similar to the process of step S220 shown in Fig. 17. Therefore, a repeated description of the process of step S510 will be omitted here.

[0129] Next, first processing unit 162 generates a moving image of the entire swing of the subject based on the motion data calculated in step S510 (step S520). The process of step S520 is the same as the process of step S230 shown in Fig. 17. Therefore, a repeated description of the process of step S520 will be omitted here.

[0130] Next, the first processing unit 162 reads out the first correspondence information and the second correspondence information stored in advance in the first storage unit 12 from the second storage unit 22. Then, the first processing unit 162 calculates the importance of each reference posture-related part of the subject U based on the read first correspondence information and second correspondence information and the motion data calculated in step S510 (step S530). The process of step S530 is the same as the process of step S240 shown in Fig. 17. Therefore, a repeated description of the process of step S530 will be omitted here.

[0131] Next, the first processing unit 162 identifies a target reference posture-related part based on the importance calculated in step S530 (step S540). The process of step S540 is the same as the process of step S250 shown in Fig. 17. Therefore, a repeated description of the process of step S540 will be omitted here.

[0132] Next, first processing unit 162 generates a partial subject swing video based on the subject person entire swing video generated in step S520 and the subject reference posture-related parts identified in step S540 (step S550). The process of step S550 is the same as the process of step S130 shown in Fig. 17. Therefore, a repeated description of the process of step S550 will be omitted here.

[0133] Then, in step S140, the display control unit 163 reads out the third correspondence information pre-stored in the first memory unit 12 from the first memory unit 12, and displays on the display unit 15, using an initial viewpoint based on the read third correspondence information, the partial motion image of the subject's swing generated by the first processing unit 162 in step S550 and the entire motion image of the subject's swing generated by the first processing unit 162 in step S520.

[0134] Through the above-described processing, the information processing system 1 also displays a moving image of the entire subject's swing and a moving image of a portion of the subject's swing. This also allows the information processing system 1 to allow the user to efficiently improve their swing.

[0135] The information processing terminal 10 described above may be integrated with the information processing device 20.

[0136] The matters described above may also be applied to other types of exercise instead of golf. For example, the information processing system 1 may be configured to assist the subject U in swinging a racket based on a video of the subject U's swing motion in a sport involving the action of hitting a ball with a racket, such as tennis or table tennis. In this case, the reference postures are, for example, the take-back, impact, and follow-through. For example, the information processing system 1 may be configured to assist the subject U in kicking a ball based on a video of the subject U's kicking motion in a sport involving the action of hitting a ball with the foot, such as soccer. In this case, the reference postures are, for example, the step-in, impact, and follow-through. For example, the information processing system 1 may be configured to assist the subject U in pitching based on a video of the subject U's pitching motion in a sport involving the action of hitting a ball with one hand, such as baseball. In this case, the reference postures are, for example, the windup, early cocking, late cocking, acceleration, and follow-through. Furthermore, for example, the information processing system 1 may be configured to assist the subject U in shooting based on a video of the subject U's shooting motion in a sport that involves using both hands to throw a ball, such as basketball. Furthermore, for example, the information processing system 1 may be configured to assist the subject U in running based on a video of the subject U's running motion in a sport that requires the subject U to repeatedly perform the same form, such as running or hurdle racing. In this case, the reference postures are, for example, the posture at the timing when the next pivot foot moves forward, the posture at the timing when the foot touches the ground, and the posture at the timing when the foot opposite the pivot foot is about to move forward. Note that, in the case of hurdle racing, the information processing system 1 may treat the posture of jumping over the hurdle as equivalent to the impact in golf, the jumping posture as equivalent to the address in golf, and the landing posture as equivalent to the follow-through in golf.In any of these cases, the scores and index values ​​described above may be increased in order of the associated part's proximity to the waist, or may be increased depending on whether the subject U's waist is lower than the waist position of the model, and weighting may be performed to achieve this.

[0137] Furthermore, the above-described contents may be combined in any manner.

[0138] <Additional Notes> [1] An information processing method comprising: calculating, based on a video of a subject performing a series of movements, a first importance obtained by multiplying a first score evaluating whether the posture of a first part of the subject in a first reference posture that serves as a basis for the series of movements is good or bad in comparison with the posture of the first part of a model person in the first reference posture by a first weight associated with the first part in the first reference posture; and calculating a second importance obtained by multiplying a second score evaluating whether the posture of a second part of the subject in a second reference posture that serves as a basis for the series of movements is good or bad in comparison with the posture of the second part of the model person in the second reference posture by a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture, and the second weight indicates the importance of improving the posture of the second part in the second reference posture. [2] The information processing method according to [1], wherein the second reference attitude is the same attitude as the first reference attitude. [3] The information processing method according to [1], wherein the second reference attitude is an attitude different from the first reference attitude. [4] The information processing method described in any one of [1] to [3], wherein the series of movements is a golf swing by the subject, and each of the first reference posture and the second reference posture is either an address, a halfway back, a top, an impact, or a follow-through. [5] An information processing method according to any one of [1] to [4], wherein an action assistance video that assists the subject in performing the series of actions is displayed on a display unit according to the calculated first importance and second importance. [6] The information processing method according to any one of [1] to [5], wherein the first weight and the second weight are determined based on a predetermined condition. [7] The information processing method according to [6], wherein the predetermined condition is that a larger weight is assigned to a posture that is executed earlier in the series of movements. [8] The information processing method according to [6], wherein the predetermined condition is that a posture included in the series of movements is assigned a larger weight as it becomes closer to a predetermined reference posture. [9] An information processing method described in any one of [1] to [8], wherein the first region is closer to the subject's waist than the second region, and the first weight is greater than the second weight.

[10] An information processing method according to any one of [1] to [9], wherein, when it is determined based on the first importance and the second importance that the importance of improving the posture of the first part of the subject is higher than the importance of improving the posture of the second part of the subject, a video showing the movement of the first part of the person is displayed on a display unit together with a video showing the movement of the first part of the subject; and when it is determined based on the first importance and the second importance that the importance of improving the posture of the second part of the subject is higher than the importance of improving the posture of the first part of the subject, a video showing the movement of the second part of the person is displayed on a display unit together with a video showing the movement of the second part of the subject.

[11] An information processing method described in

[10] , wherein the initial viewpoints of the video showing the movement of the first part of the subject and the video showing the movement of the first part of the person are predetermined, the initial viewpoints of the video showing the movement of the second part of the subject and the video showing the movement of the second part of the person are predetermined, and the initial viewpoints of the video showing the movement of the first part of the subject and the video showing the movement of the first part of the person are different from the initial viewpoints of the video showing the movement of the second part of the subject and the video showing the movement of the second part of the person.

[12] An information processing method according to any one of [1] to

[11] , wherein the calculation of the first importance and the calculation of the second importance are repeated a predetermined number of times, and based on the first importance calculated repeatedly a predetermined number of times and the second importance calculated repeatedly a predetermined number of times, it is determined whether the importance of improving the posture of the first part of the subject is higher than the importance of improving the posture of the second part of the subject.

[13] The information processing method according to any one of [1] to

[12] , wherein the first part and the second part are mutually different joints.

[14] The information processing method according to any one of [1] to

[12] , wherein at least one of the first part and the second part is a part including a plurality of joints.

[15] An information processing method according to any one of [1] to

[14] , wherein, in response to a received operation, a display unit displays an estimated motion video showing the series of motions that the subject is estimated to perform when a selected posture is improved from the posture of the first part of the subject in a selected reference posture selected from the first reference posture and the second reference posture, and the posture of the second part of the subject in the selected reference posture.

[16] A program that causes a computer to calculate, based on a video of a subject performing a series of movements, a first importance obtained by multiplying a first score that evaluates whether the posture of a first part of the subject in a first reference posture that serves as the basis for the series of movements is good or bad in comparison with the posture of the first part of a model person in the first reference posture by a first weight associated with the first part in the first reference posture; and a second importance obtained by multiplying a second score that evaluates whether the posture of a second part of the subject in a second reference posture that serves as the basis for the series of movements is good or bad in comparison with the posture of the second part of the model person in the second reference posture by a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture, and the second weight indicates the importance of improving the posture of the second part in the second reference posture.

[17] An information processing system that calculates, based on a video of a subject performing a series of movements, a first importance obtained by multiplying a first score that evaluates whether the posture of a first part of the subject in a first reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the first part of a model person in the first reference posture, and a first weight associated with the first part in the first reference posture; and a second importance obtained by multiplying a second score that evaluates whether the posture of a second part of the subject in a second reference posture that serves as a basis for the series of movements is good or bad compared to the posture of the second part of the model person in the second reference posture, and a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture, and the second weight indicates the importance of improving the posture of the second part in the second reference posture.

[18] The information processing system according to

[17] , further comprising an information processing terminal that captures the moving image.

[19] The information processing system according to

[17] or

[18] , further comprising an information processing device for acquiring the moving image.

[20] An information processing terminal that calculates, based on a video of a subject performing a series of movements, a first importance obtained by multiplying a first score evaluating whether the posture of a first part of the subject in a first reference posture that serves as a reference for the series of movements is good or bad compared to the posture of the first part of a model person in the first reference posture by a first weight associated with the first part in the first reference posture; and a second importance obtained by multiplying a second score evaluating whether the posture of a second part of the subject in a second reference posture that serves as a reference for the series of movements is good or bad compared to the posture of the second part of the model person in the second reference posture by a second weight associated with the second part in the second reference posture, wherein the first weight indicates the importance of improving the posture of the first part in the first reference posture, and the second weight indicates the importance of improving the posture of the second part in the second reference posture.

[0139] The embodiments of this disclosure have been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and may be changed, substituted, deleted, etc. as long as it does not deviate from the gist of this disclosure.

[0140] A program for implementing the functions of any of the components of the above-described device may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Here, the device in question may be, for example, an information processing terminal 10 or an information processing device 20. Note that the term "computer system" herein includes hardware such as an operating system (OS) and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and compact disks (CDs)-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0141] The above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. The program may also be a program for realizing some of the functions described above, or may be a so-called differential file or differential program that can realize the functions described above in combination with a program already recorded in the computer system. [Explanation of symbols]

[0142] 1...information processing system, 10...information processing terminal, 11...first processor, 12...first memory unit, 13...input receiving unit, 14...first communication unit, 15...display unit, 16...first control unit, 20...information processing device, 21...second processor, 22...second memory unit, 24...second communication unit, 26...second control unit, 161...imaging control unit, 162...first processing unit, 163...display control unit, 261...second processing unit, C...imaging unit

Claims

1. Based on video footage of the subject performing a series of actions, a first importance obtained by multiplying a first score that evaluates whether a posture of a first part of the subject in a first reference posture that is a reference for the series of movements is good or bad compared to a posture of the first part of a model in the first reference posture, and a first weight associated with the first part in the first reference posture; a second importance obtained by multiplying a second score that evaluates whether a posture of a second part of the subject in a second reference posture that is a reference for the series of movements is good or bad with respect to a posture of the second part of the person in the second reference posture by a second weight associated with the second part in the second reference posture; Calculate the first weight indicates the importance of improving the posture of the first part in the first reference posture, the second weight indicates the importance of improving the posture of the second part in the second reference posture; Information processing methods.

2. the second reference attitude is the same attitude as the first reference attitude; The information processing method according to claim 1 .

3. the second reference attitude is a different attitude from the first reference attitude; The information processing method according to claim 1 .

4. the series of movements is a golf swing by the subject, Each of the first reference posture and the second reference posture is any one of address, halfway back, top, impact, and follow-through. The information processing method according to claim 1 .

5. displaying, on a display unit, an action assistance moving image that assists the subject in performing the series of actions, according to the calculated first importance and second importance; The information processing method according to claim 1 .

6. The first weight and the second weight are determined based on a predetermined condition. The information processing method according to claim 1 .

7. The predetermined condition is that a larger weight is associated with a posture that is executed earlier in the series of movements. The information processing method according to claim 6.

8. The predetermined condition is that a posture included in the series of movements is assigned a larger weight as it becomes closer to a predetermined reference posture. The information processing method according to claim 6.

9. the first region is closer to a waist of the subject than the second region, The first weight is greater than the second weight. The information processing method according to claim 1 .

10. when it is determined based on the first importance and the second importance that the importance of improving the posture of the first part of the subject is higher than the importance of improving the posture of the second part of the subject, displaying a moving image showing the movement of the first part of the subject on a display unit together with a moving image showing the movement of the first part of the subject; when it is determined based on the first importance and the second importance that the importance of improving the posture of the second part of the subject is higher than the importance of improving the posture of the first part of the subject, displaying a moving image showing the movement of the second part of the subject on a display unit together with a moving image showing the movement of the second part of the subject. The information processing method according to claim 1 .

11. an initial viewpoint of each of the moving image showing the movement of the first part of the subject and the moving image showing the movement of the first part of the person is determined in advance; an initial viewpoint of each of the moving image showing the movement of the second part of the subject and the moving image showing the movement of the second part of the person is determined in advance; an initial viewpoint of each of the moving image showing the movement of the first part of the subject and the moving image showing the movement of the first part of the person is different from an initial viewpoint of each of the moving image showing the movement of the second part of the subject and the moving image showing the movement of the second part of the person; The information processing method according to claim 10.

12. repeating the calculation of the first importance and the calculation of the second importance a predetermined number of times, and determining whether or not the importance of improving the posture of the first part of the subject is higher than the importance of improving the posture of the second part of the subject, based on the first importance calculated repeatedly a predetermined number of times and the second importance calculated repeatedly a predetermined number of times. The information processing method according to claim 1 .

13. The first portion and the second portion are different joints. The information processing method according to claim 1 .

14. At least one of the first region and the second region is a region including a plurality of joints. The information processing method according to claim 1 .

15. displaying, on a display unit, an estimated movement video that shows the series of movements that the subject is estimated to make when the selected posture is improved from the posture of the first part of the subject in a selected reference posture selected from the first reference posture and the second reference posture and the posture of the second part of the subject in the selected reference posture; The information processing method according to claim 1 .

16. On the computer, Based on video footage of the subject performing a series of actions, a first importance obtained by multiplying a first score that evaluates whether a posture of a first part of the subject in a first reference posture that is a reference for the series of movements is good or bad compared to a posture of the first part of a model in the first reference posture, and a first weight associated with the first part in the first reference posture; a second importance obtained by multiplying a second score that evaluates whether a posture of a second part of the subject in a second reference posture that is a reference for the series of movements is good or bad with respect to a posture of the second part of the person in the second reference posture by a second weight associated with the second part in the second reference posture; A program for calculating the first weight indicates the importance of improving the posture of the first part in the first reference posture, the second weight indicates the importance of improving the posture of the second part in the second reference posture; program.

17. Based on video footage of the subject performing a series of actions, a first importance obtained by multiplying a first score that evaluates whether a posture of a first part of the subject in a first reference posture that is a reference for the series of movements is good or bad compared to a posture of the first part of a model in the first reference posture, and a first weight associated with the first part in the first reference posture; a second importance obtained by multiplying a second score that evaluates whether a posture of a second part of the subject in a second reference posture that is a reference for the series of movements is good or bad with respect to a posture of the second part of the person in the second reference posture by a second weight associated with the second part in the second reference posture; An information processing system for calculating the first weight indicates the importance of improving the posture of the first part in the first reference posture, the second weight indicates the importance of improving the posture of the second part in the second reference posture; Information processing system.

18. the information processing system includes an information processing terminal that captures the moving image, 18. The information processing system according to claim 17.

19. the information processing system includes an information processing device that acquires the moving image, 18. The information processing system according to claim 17.

20. Based on video footage of the subject performing a series of actions, a first importance obtained by multiplying a first score that evaluates whether a posture of a first part of the subject in a first reference posture that is a reference for the series of movements is good or bad compared to a posture of the first part of a model in the first reference posture, and a first weight associated with the first part in the first reference posture; a second importance obtained by multiplying a second score that evaluates whether a posture of a second part of the subject in a second reference posture that is a reference for the series of movements is good or bad with respect to a posture of the second part of the person in the second reference posture by a second weight associated with the second part in the second reference posture; An information processing terminal that calculates the first weight indicates the importance of improving the posture of the first part in the first reference posture, the second weight indicates the importance of improving the posture of the second part in the second reference posture; Information processing terminal.

Images