Control method, program, information processing system, and information processing apparatus

Abstract

To provide a control method capable of displaying, from a desired angle, both movement of a target person and movement of a tool in a series of actions.SOLUTION: A control method for an information processing system including an information processing terminal and an information processing apparatus causes a display unit to display a tool model based on trajectory information detected by a detection unit that detects the trajectory information indicating a trajectory of a tool used by the target person in the series of actions, together with a target person model based on a moving image obtained by capturing a series of actions performed by the target person. The target person model is a three-dimensional model indicating the series of actions performed by the target person, and the tool model is a three-dimensional model indicating the trajectory of the tool in the series of actions performed by the target person.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to a control method, a program, an information processing system, and an information processing device. [Background technology]

[0002] Patent document 1 discloses an information processing system that generates a three-dimensional stick picture showing the trajectory of a golf club based on data detected by a magnetic sensor attached to the golf club, and then synthesizes and displays the generated three-dimensional stick picture with a two-dimensional moving image capturing the golfer's swing motion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-089816 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology combines a three-dimensional stick picture with a two-dimensional moving image of a golfer's swing motion, so the combined moving image could only be displayed as a moving image seen from a single viewpoint. [Means for solving the problem]

[0005] In order to solve the above problem, one aspect of the present disclosure is a control method for an information processing system comprising an information processing terminal and an information processing device, wherein a subject model based on video images capturing a series of actions by the subject, and a tool model based on trajectory information detected by a detection unit that detects the trajectory information indicating the trajectory of a tool used by the subject in the series of actions, are displayed on a display unit, wherein the subject model is a three-dimensional model indicating the series of actions by the subject, and the tool model is a three-dimensional model indicating the trajectory of the tool in the series of actions by the subject.

[0006] In order to solve the above problem, one aspect of the present disclosure is a control method for controlling an information processing device, which causes a display unit to display, together with a subject model based on video images capturing a series of actions by the subject, a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions, wherein the subject model is a three-dimensional model indicating the series of actions by the subject, and the tool model is a three-dimensional model indicating the trajectory of the tool in the series of actions by the subject.

[0007] In order to solve the above problem, one aspect of the present disclosure is a program that causes a computer to implement the above-described control method.

[0008] In order to solve the above problem, one aspect of the present disclosure is an information processing system comprising an information processing terminal and an information processing device, and comprising a display control unit that causes a display unit to display, together with a subject model based on video images of a series of actions by the subject, a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions, wherein the subject model is a three-dimensional model indicating the series of actions by the subject, and the tool model is a three-dimensional model indicating the trajectory of the tool in the series of actions by the subject.

[0009] In order to solve the above problem, one aspect of the present disclosure is an information processing device that includes a display control unit that causes a display unit to display, together with a subject model based on video images of a series of actions by the subject, a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions, wherein the subject model is a three-dimensional model that indicates the series of actions by the subject, and the tool model is a three-dimensional model that indicates the trajectory of the tool in the series of actions by the subject. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system 1. FIG. [Figure 2] This is a diagram showing the five standard positions in a golf swing arranged in chronological order. [Figure 3] 10A and 10B are diagrams showing an example of displaying two three-dimensional models, a target person model and a tool model, by the information processing system 1. [Figure 4] 4A to 4C are diagrams showing examples of the three-dimensional model V1 and the three-dimensional model V2 displayed on a touch panel of a smartphone when a user performs a touch operation in the direction of the arrow A1 shown in FIG. 3 on the touch panel on which the three-dimensional model V1 and the three-dimensional model V2 are displayed. [Figure 5] 2 is a diagram illustrating an example of a hardware configuration of an information processing terminal 10. FIG. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device 30. [Figure 7] 2 is a diagram illustrating an example of a functional configuration of an information processing terminal 10 and an information processing device 30. FIG. [Figure 8] FIG. 10 is a diagram showing an example of a processing flow in which the information processing device 30 generates a subject model. [Figure 9] FIG. 10 is a diagram showing an example of a processing flow in which the information processing device 30 generates a tool model. [Figure 10] FIG. 10 is a diagram showing an example of a processing flow in which the information processing device 30 generates a composite three-dimensional model. [Figure 11]FIG. 10 is a diagram showing an example of a processing flow in which the information processing terminal 10 displays a composite three-dimensional model. DETAILED DESCRIPTION OF THE INVENTION

[0011] <Embodiment> Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] <Outline of the information processing system> First, an overview of an information processing system according to an embodiment will be described.

[0013] An information processing system according to an embodiment includes an information processing terminal and an information processing device. The information processing system displays, on a display unit, a subject model based on video images of a series of actions performed by the subject, as well as a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions. The subject model is a three-dimensional model indicating the series of actions performed by the subject. The tool model is also a three-dimensional model indicating the trajectory of the tool in the series of actions performed by the subject. This allows the information processing system to display both the subject's movements and the tool movements in the series of actions from a desired angle.

[0014] The configuration of the information processing system according to the embodiment and the processing performed by the information processing system will be described in detail below.

[0015] <Configuration of information processing system> The configuration of the information processing system according to the embodiment will be described below using the information processing system 1 as an example.

[0016] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system 1. As shown in FIG.

[0017] The information processing system 1 assists the subject UR in performing a series of movements based on video images of the subject UR performing a predetermined series of movements. More specifically, the information processing system 1 assists the subject UR so that the series of movements performed by the subject UR resembles the series of movements performed by a model person. Here, the series of movements is a series of movements that a person performing a predetermined type of exercise repeatedly performs in the exercise, and is a series of movements that is composed of multiple movements that the person performs consecutively. Therefore, the series of movements includes multiple predetermined reference reference postures. The following describes, as an example, a case where the exercise is golf. In this case, the series of movements is a golf swing. A golf swing includes five predetermined reference postures: address, halfway back, top, impact, and follow-through, as shown in FIG. 2. FIG. 2 is a diagram showing the five reference postures in a golf swing arranged in chronological order. The reference posture represented by A in FIG. 2 is the address. The reference posture represented by H in FIG. 2 is the halfway back. It should be noted that halfway back is sometimes referred to as half top. The reference position represented by T in FIG. 2 is the top. The reference position represented by I in FIG. 2 is the impact. The reference position represented by F in FIG. 2 is the follow-through. For ease of explanation, these five reference positions will be simply referred to as the reference positions unless there is a need to distinguish between them. For ease of explanation, a golf swing will be simply referred to as the swing in the following explanation. It should be noted that the subject UR may be any person. Therefore, the subject UR may be a user of the information processing system 1, or may not be a user of the information processing system 1.

[0018] When the subject UR makes a swing, the subject UR can make the subject UR's swing closer to that of a model person by bringing the subject UR's reference posture closer to that of the model person. This is useful for the subject UR to improve his / her golf skills. Therefore, the information processing system 1 assists the subject UR so that the subject UR's swing approaches that of the model person. The model person may be, for example, a professional golfer, a golf coach, etc., but is not limited to these. In the following, for convenience of explanation, the model person will be referred to as a target person. In other words, the information processing system 1 assists the subject UR so that the subject UR's swing approaches that of the target person.

[0019] Here, the information processing system 1 provides assistance to the subject UR by displaying a subject model based on a video of the subject UR's swing, along with a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of the tool used by the subject UR in the swing. The subject model is a three-dimensional model representing the subject UR's swing. More specifically, the subject model is a three-dimensional model that reproduces the temporal changes in the posture of the subject UR while swinging within a predetermined virtual space VS1. The virtual space VS1 is a virtual space generated within the storage area of ​​the information processing system 1. The information processing system 1 virtually places the subject model within the virtual space VS1 and displays the movement of the subject model within the virtual space VS1, thereby visualizing the subject UR's swing. In other words, the information processing system 1's display of the subject model means that the information processing system 1 displays the virtual space VS1 in which the subject model is placed. Furthermore, in the case where the series of movements is a golf swing, as in this example, the tool is a golf club. In other words, in this example, the tool trajectory refers to the change in the position of the golf club over time. Below, as an example, a case will be described in which the position of the golf club is represented by the position of the grip end of the golf club. Note that the position of the golf club may be represented by the position of a part of the golf club instead of the position of the grip end of the golf club. The tool model is a three-dimensional model that shows the trajectory of the golf club during a swing by the subject UR. More specifically, the tool model is a three-dimensional model that reproduces the change in the position of the golf club used by the subject UR during a swing within a predetermined virtual space VS2. The virtual space VS2 is a virtual space generated within the storage area of ​​the information processing system 1. The information processing system 1 virtually places the tool model within the virtual space VS2 and displays the movement of the tool model within the virtual space VS2, thereby visualizing the movement of the golf club used by the subject UR during a swing.In other words, when the information processing system 1 displays the tool model, it means that the information processing system 1 displays the virtual space VS2 in which the tool model is located. Because both the subject model and the tool model are three-dimensional models, the information processing system 1 can display both the subject model showing the movement of the subject UR in a swing and the tool model showing the movement of the golf club in the swing by the subject UR from an angle desired by the subject UR. As a result, the information processing system 1 can assist the subject UR in making his or her swing closer to that of the target person. The subject model is a three-dimensional model generated by the information processing system 1. As long as it is a three-dimensional model showing the swing by the subject UR, it may be a three-dimensional model using an avatar of the subject UR, a three-dimensional model showing the trajectory of the movement of joint points, or another type of three-dimensional model. As long as it is a three-dimensional model showing the trajectory of the golf club, it may be a three-dimensional model using an avatar of the subject UR, a three-dimensional model showing the trajectory of the golf club, or another type of three-dimensional model.

[0020] The information processing system 1 may be configured to display the subject model and the tool model separately, or may be configured to display the subject model and the tool model together in an overlapping manner. As an example, the case where the information processing system 1 displays the subject model and the tool model together in an overlapping manner will be described below.

[0021] When displaying such three-dimensional models, the information processing system 1 synchronizes the time of the two three-dimensional models, synchronizes the three-dimensional coordinate systems of the two three-dimensional models, adjusts the scale of the two three-dimensional models, etc. This allows the information processing system 1 to prevent the two three-dimensional models, the subject model and the tool model, that are displayed superimposed on each other from being out of sync in time and space.

[0022] Fig. 3 is a diagram showing an example of displaying two three-dimensional models, a subject model and a tool model, by the information processing system 1. In the example shown in Fig. 3, the information processing system 1 displays these two three-dimensional models together, superimposed on the touch panel of a smartphone.

[0023] The three-dimensional model V1 shown in Fig. 3 is an example of a subject model. That is, in the example shown in Fig. 3, the three-dimensional model V1, which is a subject model, is a three-dimensional model using an avatar of the subject UR. On the other hand, the three-dimensional model V2 shown in Fig. 3 is an example of a tool model. That is, in this example, the three-dimensional model V2, which is a tool model, is a three-dimensional model that shows the trajectory of a golf club.

[0024] Here, for example, when a user performs a touch operation in the direction indicated by the arrow A1 in FIG. 3 on a touch panel of a smartphone on which three-dimensional models V1 and V2 are displayed, the information processing system 1 displays the three-dimensional models V1 and V2 on the touch panel as viewed from an angle corresponding to the received touch operation. In other words, the information processing system 1 changes the orientation of the subject model and the tool model displayed on the touch panel in response to the received touch operation. This allows the information processing system 1 to display both the movement of the subject UR during a swing and the movement of the golf club from an angle desired by the user. FIG. 4 is a diagram illustrating an example of the three-dimensional models V1 and V2 displayed on the touch panel when the user performs a touch operation in the direction indicated by the arrow A1 in FIG. 3 on the touch panel of a smartphone on which three-dimensional models V1 and V2 are displayed. As shown in FIG. 4, when the touch operation is performed on the touch panel, the information processing system 1 changes the orientation of the subject model and the tool model displayed on the touch panel.

[0025] The information processing system 1 includes, for example, an information processing terminal 10, a detection unit 20, and an information processing device 30. The information processing system 1 may not include the detection unit 20. In the information processing system 1, the information processing terminal 10 is communicatively connected to the information processing device 30 wirelessly or via a wire. In the example shown in FIG. 1, the information processing terminal 10 is communicatively connected to the information processing device 30 wirelessly or via a wire. In the information processing system 1, the detection unit 20 is communicatively connected to the information processing device 30 wirelessly or via a wire. In the example shown in FIG. 1, the detection unit 20 is communicatively connected to the information processing device 30 wirelessly. In the information processing system 1, the information processing terminal 10 may be communicatively connected to the detection unit 20 wirelessly or via a wire.

[0026] The information processing terminal 10 is, for example, a multi-function mobile phone terminal (smartphone), but is not limited to this and may be other information processing terminals such as a mobile phone terminal, a tablet PC (Personal Computer), a PDA (Personal Digital Assistant), or a notebook PC.

[0027] The information processing terminal 10 includes an imaging unit C that can capture moving images. Note that the information processing terminal 10 may not include the imaging unit C, and may instead be configured to be connected to the imaging unit C as an external device.

[0028] The imaging unit C is a camera equipped with an imaging element such as a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like that converts collected light into an electrical signal. In this example, the imaging unit C is built into the information processing terminal 10. Therefore, the imaging unit C moves in accordance with the movement of the information processing terminal 10. In other words, the range that the imaging unit C can capture changes in accordance with the movement of the information processing terminal 10. The imaging unit C captures two-dimensional moving images of that range. Note that the imaging unit C may also be configured to be capable of capturing still images of that range.

[0029] In response to the received operation, the information processing terminal 10 causes the imaging unit C to capture a moving image within the range that the imaging unit C can capture. This allows the information processing terminal 10 to capture a moving image of the subject UR performing a swing. While capturing the moving image, the information processing terminal 10 transmits the captured frames of the moving image in order to the information processing device 30. The information processing terminal 10 continues transmitting such frames until capturing of the moving image is completed.

[0030] After completing transmission of the frame to the information processing device 30, the information processing terminal 10 waits until it receives, as a response from the information processing device 30, video data of a three-dimensional model in which a tool model is superimposed on a subject model. Hereinafter, for convenience of explanation, the three-dimensional model will be referred to as a composite three-dimensional model, and the video data will be referred to as composite video data. When the information processing terminal 10 receives the composite video data, it displays a composite three-dimensional model of the received composite video data. This allows the information processing terminal 10 to display both the movement of the subject person UR and the movement of the golf club during a swing from a desired angle. After displaying the composite three-dimensional model, the information processing terminal 10 changes the orientation of the displayed composite three-dimensional model in accordance with the received operation.

[0031] The detection unit 20 is an inertial measurement device that detects the angular velocity and acceleration occurring in the detection unit 20. The detection unit 20 is attached, for example, to the grip end of a golf club. This allows the detection unit 20 to detect the angular velocity and acceleration occurring in the detection unit 20 as the angular velocity and acceleration of the golf club. The detection unit 20 outputs measurement data indicating the detected angular velocity and acceleration to the information processing device 30.

[0032] The information processing device 30 may be any information processing device that can function as a server, such as a workstation, a desktop PC, or a notebook PC, but is not limited to these.

[0033] The information processing device 30 waits until it starts receiving frames of a moving image from the information processing terminal 10. When the information processing device 30 starts receiving frames of a moving image, it determines whether the posture of the subject UR is at address based on the frames received in order. If the information processing device 30 determines that the posture of the subject UR is at address, it starts receiving measurement data from the detection unit 20. After receiving all frames of the moving image, the information processing device 30 stops receiving measurement data from the detection unit 20 and generates a subject model based on all the received frames. The method for generating the subject model based on all the frames may be a known method based on inverse kinematics or the like, or a method to be developed in the future. The information processing device 30 generates a tool model based on all the measurement data received from the detection unit 20. The method for generating the tool model based on all the measurement data may be a known method based on inverse kinematics or the like, or a method to be developed in the future.

[0034] The information processing device 30 generates two 3D models, a subject model and a tool model, and then generates a composite 3D model by overlaying the tool model on the subject model. At this time, the information processing device 30 synchronizes the time of the two 3D models, synchronizes the 3D coordinate systems of the two 3D models, adjusts the scale of the two 3D models, etc. This allows the information processing system 1 to prevent temporal and spatial misalignment between the two 3D models, the subject model and the tool model, which are displayed superimposed. After generating the composite 3D model, the information processing device 30 transmits video data of the generated composite 3D model to the information processing terminal 10 as a response to a frame received from the information processing terminal 10.

[0035] <Hardware configuration of information processing terminal> The hardware configuration of the information processing terminal 10 will be described below with reference to Fig. 5. Fig. 5 is a diagram showing an example of the hardware configuration of the information processing terminal 10.

[0036] The information processing terminal 10 includes, for example, a first processor 11, a first storage unit 12, an input receiving unit 13, a first communication unit 14, a display unit 15, and an imaging unit C. These components are connected to each other via a bus so that they can communicate with each other. The information processing terminal 10 also communicates with the information processing device 30 via the first communication unit 14.

[0037] The first processor 11 is, for example, a CPU (Central Processing Unit). Note that the first processor 11 may be another processor such as an FPGA (Field Programmable Gate Array) instead of a CPU. The first processor 11 executes various programs stored in the first storage unit 12.

[0038] The first storage unit 12 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), etc. Note that the first storage unit 12 may be an external storage device connected via a digital input / output port such as a universal serial bus (USB) instead of being built into the information processing terminal 10. The first storage unit 12 stores various types of information, images, programs, etc. that are processed by the information processing terminal 10.

[0039] The input receiving unit 13 is an input device, and is, for example, a touch panel that is configured integrally with the display unit 15. Note that the input receiving unit 13 may be another input device, such as a keyboard, a mouse, or a touchpad, that is configured separately from the display unit 15, instead of the touch panel.

[0040] The first communication unit 14 includes, for example, a digital input / output port such as a USB, an Ethernet (registered trademark) port, an antenna for wireless communication, and the like.

[0041] The display unit 15 is a display device, and includes, for example, a liquid crystal display.

[0042] <Hardware configuration of information processing device> The hardware configuration of the information processing device 30 will be described below with reference to Fig. 6. Fig. 6 is a diagram showing an example of the hardware configuration of the information processing device 30.

[0043] The information processing device 30 includes, for example, a second processor 31, a second storage unit 32, and a second communication unit 34. These components are connected to each other via a bus so that they can communicate with each other. The information processing device 30 also communicates with the information processing terminal 10 via the second communication unit 34.

[0044] The second processor 31 is, for example, a CPU. Note that the second processor 31 may be another processor such as an FPGA instead of a CPU. The second processor 31 executes various programs stored in the second storage unit 32.

[0045] The second storage unit 32 includes, for example, an HDD, an SSD, an EEPROM, a ROM, a RAM, etc. Note that the second storage unit 32 may be an external storage device connected via a digital input / output port such as a USB, instead of being built into the information processing device 30. The second storage unit 32 stores various types of information, various images, various programs, etc. that are processed by the information processing device 30.

[0046] The second communication unit 34 includes, for example, a digital input / output port such as a USB, an Ethernet (registered trademark) port, an antenna for wireless communication, and the like.

[0047] <Functional configuration of information processing terminal and information processing device> The functional configurations of the information processing terminal 10 and the information processing device 30 will be described below with reference to Fig. 7. Fig. 7 is a diagram showing an example of the functional configurations of the information processing terminal 10 and the information processing device 30.

[0048] The information processing terminal 10 includes a first storage unit 12, an input receiving unit 13, a first communication unit 14, a display unit 15, an imaging unit C, and a first control unit 16.

[0049] The first control unit 16 controls the entire information processing terminal 10. The first control unit 16 includes an imaging control unit 161, a first processing unit 162, and a display control unit 163. These functional units included in the first control unit 16 are realized, for example, by the first processor 11 executing various programs stored in the first storage unit 12. Some or all of the functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

[0050] The imaging control unit 161 controls the imaging unit C.

[0051] The first processing unit 162 performs various processes performed by the information processing terminal 10.

[0052] The display control unit 163 generates various images in response to operations received via the input receiving unit 13. The display control unit 163 displays the generated images on the display unit 15. Furthermore, the display control unit 163 displays moving images of moving image data received from the information processing device 30 on the display unit 15 in response to operations received via the input receiving unit 13.

[0053] The information processing device 30 includes a second storage unit 32, a second communication unit , and a second control unit .

[0054] The second control unit 36 ​​controls the entire information processing device 30. The second control unit 36 ​​includes a second processing unit 361. The second processing unit 361 included in the second control unit 36 ​​is realized, for example, by the second processor 31 executing various programs stored in the second storage unit 32. The second processing unit 361 may also be a hardware functional unit such as an LSI or an ASIC.

[0055] The second processing unit 361 performs various processes performed by the information processing device 30.

[0056] <Processing by which an information processing device generates a target person model> Hereinafter, a process in which the information processing device 30 generates a subject model will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the flow of a process in which the information processing device 30 generates a subject model.

[0057] The second processing unit 361 waits until a predetermined first start condition is satisfied (step S110). Here, the first start condition is, for example, that the information processing device 30 receives from the information processing terminal 10 a request to receive frames of a moving image captured by the information processing terminal 10. However, the moving image is a moving image of the subject UR performing a swinging motion. Note that the first start condition may instead be another condition that triggers the information processing device 30 to start receiving the frames.

[0058] If the second processing unit 361 determines that the first start condition is satisfied (step S110-YES), the second processing unit 361 starts receiving frames transmitted from the information processing terminal 10 (step S120).

[0059] Next, the second processing unit 361 continues the reception of frames that started in step S120 until a predetermined first end condition is satisfied (step S130). Here, the first end condition is, for example, that the information processing device 30 receives, from the information processing terminal 10, a request to end reception of frames of moving images captured by the information processing terminal 10. Note that the first end condition may instead be another condition that triggers the information processing device 30 to end reception of the frames.

[0060] If the second processing unit 361 determines that the first end condition is satisfied (step S130-YES), it ends reception of frames transmitted from the information processing terminal 10 (step S140).

[0061] Next, the second processing unit 361 calculates a time series of three-dimensional positions for each joint of the subject person UR who is swinging, based on all frames received from the information processing terminal 10 in steps S120 to S140 (step S150). In Fig. 8, the process of step S150 is indicated by "calculation of time series of three-dimensional positions."

[0062] Next, the second processing unit 361 generates a subject model based on the time series calculated in step S150 (step S160), and ends the processing of the flowchart shown in FIG.

[0063] As described above, the information processing device 30 can receive frames of a moving image captured by the information processing terminal 10 from the information processing terminal 10, and generate a subject model based on the received frames.

[0064] <Processing by an information processing device to generate a tool model> Hereinafter, a process in which the information processing device 30 generates a tool model will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the flow of the process in which the information processing device 30 generates a tool model. As an example, the following describes a case in which the process of the flowchart shown in Fig. 8 is started at a timing before the process of step S210 shown in Fig. 9 is performed.

[0065] The second processing unit 361 waits until a predetermined second start condition is satisfied (step S210). Here, the second start condition is, for example, that the posture of the subject UR is determined to be address based on a frame acquired from the information processing terminal 10 by the processing of the flowchart shown in FIG. 8. Note that the second start condition may instead be another condition that triggers the information processing device 30 to start receiving the measurement data detected by the detection unit 20. This determination is made by the second processing unit 361 during the period in which the processing of steps S120 to S140 shown in FIG. 8 is being performed.

[0066] When the second processing unit 361 determines that the second start condition is satisfied (step S210—YES), it resets the detection unit 20. Then, the second processing unit 361 causes the detection unit 20 to start transmitting measurement data, and starts receiving measurement data transmitted from the detection unit 20 (step S220). In FIG. 9, the processing of step S220 is indicated by “start receiving measurement data.” Such resetting of the detection unit 20 may be omitted in step S220. However, by resetting the detection unit 20 in step S220, the information processing device 30 can reduce the discrepancy between the actual movement of the golf club and the detection result by the detection unit 20.

[0067] Next, the second processing unit 361 continues receiving the measurement data that started in step S220 until a predetermined second end condition is satisfied (step S230). Here, the second end condition is, for example, that reception of frames from the information processing terminal 10 has ended in the processing of the flowchart shown in Fig. 8. Note that the second end condition may instead be another condition that triggers the end of reception of the measurement data that started in step S220.

[0068] When the second processing unit 361 determines that the second termination condition is satisfied (step S230-YES), it terminates reception of the measurement data transmitted from the detection unit 20 and causes the detection unit 20 to terminate transmission of the measurement data (step S240). In Fig. 9, the processing of step S240 is indicated by "end of reception of measurement data".

[0069] Next, the second processing unit 361 calculates a time series of the three-dimensional positions of the golf club based on all of the measurement data received from the detection unit 20 in steps S220 to S240. Then, the second processing unit 361 performs interpolation processing on the calculated time series (step S240). In FIG. 9, the processing of step S240 is indicated by "interpolation processing." Here, the time series of the three-dimensional positions of the golf club calculated based on all of the measurement data is a collection of discontinuous three-dimensional positions. The interpolation processing performed by the second processing unit 361 in step S240 is processing to interpolate three-dimensional positions between three-dimensional positions that are adjacent in time in such a collection of discontinuous three-dimensional positions. The method by which the second processing unit 361 performs such interpolation processing may be a known method or a method to be developed in the future.

[0070] Next, the second processing unit 361 generates an implement model based on the time series of the three-dimensional position of the golf club after the interpolation process has been performed in step S240 (step S260), and ends the process of the flowchart shown in FIG.

[0071] As described above, the information processing device 30 can receive measurement data from the detection unit 20 and generate a tool model based on the received measurement data.

[0072] <Processing by an information processing device to generate a synthetic 3D model> Hereinafter, a process of generating a composite 3D model by the information processing device 30 will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the flow of a process of generating a composite 3D model by the information processing device 30. For example, when the information processing device 30 has completed both the generation of a subject model by the process of the flowchart shown in Fig. 8 and the generation of a tool model by the process of the flowchart shown in Fig. 9, the information processing device 30 starts the process of the flowchart shown in Fig. 10.

[0073] After both the generation of the subject model by the processing of the flowchart shown in FIG. 8 and the generation of the tool model by the processing of the flowchart shown in FIG. 9 are completed, the second processing unit 361 synchronizes the time of the generated subject model with the time of the generated tool model (step S310). In FIG. 10, the processing of step S310 is indicated by "time synchronization." For example, the second processing unit 361 synchronizes the time of the generated subject model with the time of the generated tool model so that the timing at which the posture of the subject UR is determined to be in address based on the frame acquired from the information processing terminal 10 by the processing of the flowchart shown in FIG. 8 coincides with the timing at which the processing of step S220 shown in FIG. 9 was performed. Note that the method by which the second processing unit 361 synchronizes the time of the subject model with the time of the tool model in this manner may be a known method or a method to be developed in the future.

[0074] Next, the second processing unit 361 synchronizes the three-dimensional coordinate system of the generated target person model with the three-dimensional coordinate system of the generated tool model (step S320). In Fig. 10, the processing of step S320 is indicated by "synchronization of three-dimensional coordinate systems." Here, the processing of step S320 will be described.

[0075] The second processing unit 361 performs the processing described below as the processing of step S320. The second processing unit 361 generates a matrix M indicating a point cloud in which points indicating the position of a tool model are arranged in chronological order, and a matrix A indicating a point cloud in which points indicating the position of the hand of the subject model are arranged in chronological order. Here, on the one hand, the point cloud indicated by matrix M is a collection of points indicating the position of the tool model in the virtual space VS2. Matrix M is a matrix with 3 rows and m columns. More specifically, matrix M is a matrix in which three-dimensional position vectors indicating the position of each point of the point cloud indicated by matrix M are arranged in chronological order. In other words, matrix M is a matrix in which three-dimensional coordinates indicating the position of each point of the point cloud indicated by matrix M are arranged in chronological order. Therefore, m is the number of points in the point cloud indicated by matrix M, and is the number of positions identified by the second processing unit 361 as the position of the tool model. On the other hand, the point cloud indicated by matrix A is a collection of points indicating the position of the hand of the subject model in the virtual space VS1. Matrix A is a matrix with 3 rows and a columns. More specifically, matrix A is a matrix in which three-dimensional position vectors indicating the position of each point in the point cloud indicated by matrix A are arranged in chronological order. In other words, matrix A is a matrix in which three-dimensional coordinates indicating the position of each point in the point cloud indicated by matrix A are arranged in chronological order. Therefore, a is the number of points in the point cloud indicated by matrix A, and is the number of positions identified by the second processing unit 361 as the positions of the hands of the subject model.

[0076] The position of the subject model's hand at a certain time should match or nearly match the position of the tool model at that time. This is because the position of the tool model is represented by the position of the grip end of the golf club. Given this, the second processing unit 361 can match or nearly match the position of the subject model's hand at that time with the position of the tool model at that time by matching the position of the origin and the direction of each coordinate axis of the three-dimensional coordinate system in the virtual space VS2 with the position of the origin and the direction of each coordinate axis of the three-dimensional coordinate system in the virtual space VS1. The position of the origin and the direction of each coordinate axis of the three-dimensional coordinate system in the virtual space VS2 can be matched with the position of the origin and the direction of each coordinate axis of the three-dimensional coordinate system in the virtual space VS1 by at least one of rotation and translation. This means that the matrix M and the matrix A are related by the following equation (1):

[0077]

number

[0078] In the above formula (1), R represents a rotation matrix that rotates the point group represented by matrix M. Furthermore, t in formula (1) represents a translation matrix that translates the point group represented by matrix M. Such rotation matrix R and translation matrix t can be calculated if points that correspond to at least three points included in the point group represented by matrix M can be identified from among the points included in the point group represented by matrix A. However, at this stage, it is unknown which points included in the point group represented by matrix A correspond to at least three points included in the point group represented by matrix M.

[0079] Therefore, the second processing unit 361 identifies three vectors from the three-dimensional position vectors included in the matrix A: a three-dimensional position vector when the posture of the subject UR is at address, a three-dimensional position vector when the posture of the subject UR is halfway back, and a three-dimensional position vector when the posture of the subject UR is at top. Hereinafter, for the sake of convenience, the three-dimensional position vector when the posture of the subject UR is at address will be referred to as the 11th vector. Hereinafter, for the sake of convenience, the three-dimensional position vector when the posture of the subject UR is halfway back will be referred to as the 12th vector. Hereinafter, for the sake of convenience, the three-dimensional position vector when the posture of the subject UR is at top will be referred to as the 13th vector. The method for identifying these three vectors may be, for example, a method in which the three-dimensional position vector indicating the lowest position of the tool model is identified as the 11th vector, the three-dimensional position vector indicating the rearmost position of the tool model is identified as the 12th vector, and the three-dimensional position vector indicating the highest position of the tool model is identified as the 13th vector. Alternatively, a different known method or a method to be developed may be used. Note that instead of identifying the three vectors, the 11th to 13th vectors, using a combination of the three reference postures of address, halfway back, and top, the second processing unit 361 may identify the three vectors using another combination of the three reference postures. However, even in this case, the second processing unit 361 uses the other combination so that the chronological order of the 11th, 12th, and 13th vectors does not change.

[0080] The second processing unit 361 calculates the Euclidean distance between the 11th vector and the 12th vector as the first distance. The second processing unit 361 also calculates the Euclidean distance between the 11th vector and the 13th vector as the second distance. The second processing unit 361 then identifies the three-dimensional position vector when the posture of the subject UR is at address from among the three-dimensional position vectors included in the matrix M. Hereinafter, for ease of explanation, this three-dimensional position vector will be referred to as the 21st vector. The method for identifying the 21st vector may be a method based on the movement of the subject model, a different method from this method, or a method to be developed in the future.

[0081] Based on the identified 21st vector and the calculated first distance, the second processing unit 361 identifies, from among the three-dimensional position vectors included in matrix M, a 22nd vector, which is a three-dimensional position vector when the posture of the subject UR is halfway back. More specifically, the second processing unit 361 identifies, from among the three-dimensional position vectors included in matrix M, a three-dimensional position vector whose distance from the identified 21st vector is closest to the first distance, and sets the identified three-dimensional position vector as the 22nd vector. Furthermore, based on the identified 21st vector and the calculated second distance, the second processing unit 361 identifies, from among the three-dimensional position vectors included in matrix M, a 23rd vector, which is a three-dimensional position vector when the posture of the subject UR is at the top. More specifically, the second processing unit 361 identifies, from among the three-dimensional position vectors included in matrix M, a three-dimensional position vector whose distance from the identified 21st vector is closest to the second distance, and sets the identified three-dimensional position vector as the 23rd vector. As a result, the information processing device 30 can set, for example, the 22nd vector and the 23rd vector, which cannot be identified by the subject model alone, from among the three-dimensional position vectors included in the matrix M. Furthermore, even if the time of the subject model and the time of the tool model are not perfectly synchronized, the information processing device 30 can set the 22nd vector and the 23rd vector, which are estimated to have the highest correlation. This is useful because it leads to highly accurate synchronization between the three-dimensional coordinate system of the subject model and the three-dimensional coordinate system of the tool model.

[0082] When the 22nd vector and the 23rd vector are set, the second processing unit 361 can use the 21st vector as the vector corresponding to the 11th vector, the 22nd vector as the vector corresponding to the 12th vector, and the 23rd vector as the vector corresponding to the 13th vector. In this case, the second processing unit 361 generates a matrix P M and the matrix P in which the 21st to 23rd vectors are arranged in chronological order. A can be defined as the following equation (2):

[0083]

number

[0084] In the above formula (2), s indicates the address. Also, in formula (2), h indicates the halfway back. Also, in formula (2), t indicates the top.

[0085] Furthermore, the second processing unit 361 calculates a three-dimensional position vector centroid that indicates the position of the center of gravity of the position indicated by the 11th vector, the position indicated by the 12th vector, and the position indicated by the 13th vector. M Furthermore, the second processing unit 361 calculates a three-dimensional position vector centroid that indicates the position of the center of gravity of the position indicated by the 21st vector, the position indicated by the 22nd vector, and the position indicated by the 23rd vector. A From these, the second processing unit 361 can calculate the covariance matrix H as follows:

[0086]

number

[0087] Here, (P M -centroid M ) is the vector centroid from each column of the matrix PM. M In addition, (P A -centroid A ) is the matrix P A From each column of the vector centroid A This shows the matrix after subtracting

[0088] The second processing unit 361 can calculate the orthogonal matrix U, the diagonal matrix S, and the orthogonal matrix V by performing singular value decomposition of the covariance matrix H as shown in the following equation (4). Note that SVD(H) in equation (4) represents the singular value decomposition of the covariance matrix H.

[0089]

number

[0090] Then, the second processing unit 361 can calculate the rotation matrix R as shown in the following equation (5) using the orthogonal matrix U and the orthogonal matrix V obtained by the above equation (4).

[0091]

number

[0092] Once the rotation matrix R is calculated, the translation matrix t is calculated using the matrix P M , matrix P A Based on the above formula (1), it can be calculated using the following formula (6).

[0093]

number

[0094] However, the above equation (6) is a vector centroid that indicates the position of the center of gravity. M and vector centroid A Using this, it can be approximated as in the following equation (7).

[0095]

number

[0096] Then, the above equation (7) can be solved for the translation matrix t as in the following equation (8).

[0097]

number

[0098] In this way, the second processing unit 361 can calculate the rotation matrix R and the translation matrix t. Then, the second processing unit 361 synchronizes the three-dimensional coordinate system of the subject model with the three-dimensional coordinate system of the tool model using the calculated rotation matrix R and translation matrix t. Note that the method of synchronizing the three-dimensional coordinate system of the subject model with the three-dimensional coordinate system of the tool model using the rotation matrix R and the translation matrix t may be a known method or a method to be developed in the future. Furthermore, the rotation matrix R and the translation matrix t may be derived using a registration algorithm such as ICP (Iterative Closest Point) or CPD (Coherent Point Drift), a machine learning model, or the like. The second processing unit 361 performs the above-mentioned processing as the processing of step S320.

[0099] The second processing unit 361 may be configured to adjust the scales of the subject model and the tool model in step S320. In this case, the second processing unit 361 may adjust the scales of the subject model and the tool model by any method.

[0100] After the process of step S320 is performed, the second processing unit 361 generates a composite 3D model by combining two 3D models, the subject model and the tool model, which are synchronized in time and three-dimensional coordinates (step S330). The method for generating the composite 3D model in step S330 may be a known method or a method to be developed in the future.

[0101] Next, the second processing unit 361 transmits the moving image data of the composite 3D model generated in step S330 to the information processing terminal 10 (step S340), and the process of the flowchart shown in FIG. 10 ends.

[0102] As described above, the information processing device 30 can generate a composite three-dimensional model. As a result, the information processing device 30 can display the tool model together with the subject model on the information processing terminal 10. In other words, the information processing device 30 can display both the movement of the subject person UR and the movement of the golf club during the swing from a desired angle on the information processing terminal 10.

[0103] <Processing by which an information processing terminal displays a synthetic 3D model> Hereinafter, a process in which the information processing terminal 10 displays a composite three-dimensional model will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the flow of a process in which the information processing terminal 10 displays a composite three-dimensional model. Below, as an example, a case will be described in which the information processing terminal 10 receives a process start operation to cause the information processing terminal 10 to start displaying a composite three-dimensional model at a timing before the processing of step S410 shown in Fig. 11 is performed. Also, below, as an example, a case will be described in which the information processing terminal 10 is positioned so that the target person UR is included in the range that can be imaged by the imaging unit C at that timing.

[0104] After accepting the processing start operation, the imaging control unit 161 transmits to the information processing device 30 a request for the information processing device 30 to receive frames of the moving image captured by the information processing terminal 10. Then, the imaging control unit 161 starts capturing images by the imaging unit C and transmitting the frames of the moving image captured by the imaging unit C in chronological order to the information processing device 30 (step S410). In FIG. 11, the processing of step S410 is indicated by "start capturing images and transmitting frames."

[0105] Next, the imaging control unit 161 continues the imaging and frame transmission that started in step S410 until an operation to end imaging by the imaging unit C is received (step S420). In Fig. 11, the processing of step S420 is indicated by "End of imaging?"

[0106] When the imaging control unit 161 determines that an operation to end imaging by the imaging unit C has been received (step S420-YES), it ends imaging by the imaging unit C and the transmission of frames of the moving image captured by the imaging unit C in chronological order to the information processing device 30 (step S430). Then, the imaging control unit 161 transmits to the information processing device 30 a request to end reception by the information processing device 30 of frames of the moving image captured by the information processing terminal 10. In FIG. 11, the processing of step S430 is indicated by "end imaging and frame transmission."

[0107] Here, during the period in which imaging is being performed by the imaging unit C in steps S410 to S430, the subject UR swings within a range that can be imaged by the imaging unit C. This allows the information processing device 30 to generate a subject model by the processing of the flowchart shown in FIG.

[0108] After the process of step S430 is performed, the display control unit 163 waits until it receives a composite three-dimensional model from the information processing device 30 (step S440).

[0109] When the display control unit 163 determines that a composite 3D model has been received from the information processing device 30 (step S440—YES), it causes the display unit 15 to display the composite 3D model received from the information processing device 30 (step S450). At this time, the display control unit 163 sets the orientation of the composite 3D model to be displayed on the display unit 15 to a predetermined initial orientation, and causes the display unit 15 to display the composite 3D model.

[0110] Next, the display control unit 163 waits until an operation is received (step S460).

[0111] If it is determined that an operation has been received (step S460-YES), the display control unit 163 determines whether the operation received in step S460 is an operation to end the display of the composite 3D model on the display unit 15 (step S470). In Fig. 11, the processing of step S470 is indicated by "End display of composite 3D model?"

[0112] If the display control unit 163 determines that the operation received in step S460 is an operation to end the display of the composite three-dimensional model on the display unit 15 (step S470-YES), it ends the display of the composite three-dimensional model on the display unit 15 and terminates the processing of the flowchart shown in Figure 11.

[0113] On the other hand, if the display control unit 163 determines that the operation received in step S460 is not an operation to end the display of the composite 3D model on the display unit 15 (step S470-NO), it performs processing according to the operation received in step S460 (step S480). For example, if the operation received in step S460 is an operation to change the orientation of the composite 3D model to be displayed on the display unit 15, the display control unit 163 changes the orientation of the composite 3D model displayed on the display unit 15 in accordance with the operation. After the processing of step S480 is performed, the display control unit 163 transitions to step S460 and waits again until an operation is received.

[0114] As described above, the information processing terminal 10 can display a composite three-dimensional model. That is, the information processing terminal 10 can display both the movement of the subject person UR during a swing and the movement of the golf club from a desired angle.

[0115] Note that some or all of the functions of the information processing device 30 described above may be configured integrally with the information processing terminal 10. When all of the functions of the information processing device 30 are configured integrally with the information processing terminal 10, the information processing system 1 is the information processing terminal 10 itself.

[0116] Furthermore, the above-described contents may be combined in any manner.

[0117] <Additional Notes> [1] A control method for an information processing system comprising an information processing terminal and an information processing device, the control method comprising: displaying on a display unit a subject model based on moving images capturing a series of actions by the subject, together with a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions, the subject model being a three-dimensional model indicating the series of actions by the subject, and the tool model being a three-dimensional model indicating the trajectory of the tool in the series of actions by the subject. [2] The control method described in [1] includes acquiring the moving image, generating the subject model based on the acquired moving image, acquiring the trajectory information, and generating the tool model based on the acquired trajectory information. [3] a first distance between the three-dimensional coordinates of the tool model in a first reference posture among a plurality of reference postures that serve as references for the series of movements and the three-dimensional coordinates of the tool model in a second reference posture among the plurality of reference postures that is different from the first reference posture; and a second distance between the three-dimensional coordinates of the tool model in the second reference posture and the three-dimensional coordinates of the tool model in a third reference posture among the plurality of reference postures that is different from each of the first reference posture and the second reference posture. The control method described in [2] includes setting three-dimensional coordinates indicating the position of the hand of a subject model, setting three-dimensional coordinates indicating the position of the hand of the subject model in the third reference posture based on the three-dimensional coordinates indicating the position of the hand of the subject model in the first reference posture and the second distance, and synchronizing the three-dimensional coordinate system of the subject model with the three-dimensional coordinate system of the tool model based on three coordinates: the three-dimensional coordinates indicating the position of the hand of the subject model in the first reference posture, the three-dimensional coordinates indicating the position of the hand of the subject model in the second reference posture, and the three-dimensional coordinates indicating the position of the hand of the subject model in the third reference posture. [4] The control method described in [3], which calculates the center of gravity of the three coordinates, calculates a rotation matrix that rotates the three-dimensional coordinate system of the subject model based on the calculated center of gravity, and synchronizes the three-dimensional coordinate system of the subject model and the three-dimensional coordinate system of the tool model based on the calculated rotation matrix. [5] a control method according to [4], comprising: identifying, from a time series of three-dimensional coordinates indicating the position of the hand of the subject model, a three-dimensional coordinate whose distance from the three-dimensional coordinate indicating the position of the hand of the subject model in the first reference posture is closest to the first distance; identifying the identified three-dimensional coordinate as a first three-dimensional coordinate indicating the position of the hand of the subject model in the second reference posture; identifying, from the time series of three-dimensional coordinates indicating the position of the hand of the subject model, a three-dimensional coordinate whose distance from the three-dimensional coordinate indicating the position of the hand of the subject model in the first reference posture is closest to the second distance; identifying the identified three-dimensional coordinate as a second three-dimensional coordinate indicating the position of the hand of the subject model in the third reference posture; and synchronizing a three-dimensional coordinate system of the subject model and a three-dimensional coordinate system of the tool model based on the identified first and second three-dimensional coordinates. [6] A control method as described in [5], which determines whether the subject's posture has reached the first reference posture based on the acquired moving image, and starts acquiring the trajectory information if it is determined that the subject's posture has reached the first reference posture. [7] The control method described in [6], wherein the detection unit detects the angular velocity and acceleration of the tool, and the control method resets the detection unit when it determines that the subject's posture has become the first reference posture. [8] The control method described in [7], wherein the series of movements are golf movements, the first reference position is address, the second reference position is halfway back, and the third reference position is top. [9] A control method according to any one of [1] to [8], wherein the orientation of each of the subject model and the tool model displayed on the display unit is changed in accordance with the received operation.

[10] A control method for controlling an information processing device, comprising: displaying on a display unit a subject model based on video images of a series of actions by the subject, together with a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions; wherein the subject model is a three-dimensional model indicating the series of actions by the subject, and the tool model is a three-dimensional model indicating the trajectory of the tool in the series of actions by the subject.

[11] A program that causes a computer to realize the control method described in

[10] .

[12] An information processing system comprising an information processing terminal and an information processing device, the information processing system further comprising a display control unit that causes a display unit to display, together with a subject model based on moving images capturing a series of actions by the subject, a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions, wherein the subject model is a three-dimensional model indicating the series of actions by the subject, and the tool model is a three-dimensional model indicating the trajectory of the tool in the series of actions by the subject.

[13] An information processing device comprising: a display control unit that causes a display unit to display, together with a subject model based on video images of a series of actions by the subject, a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating the trajectory of a tool used by the subject in the series of actions, wherein the subject model is a three-dimensional model indicating the series of actions by the subject, and the tool model is a three-dimensional model indicating the trajectory of the tool in the series of actions by the subject.

[0118] The embodiments of this disclosure have been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and may be changed, substituted, deleted, etc. as long as it does not deviate from the gist of this disclosure.

[0119] Furthermore, a program for implementing the functions of any of the components of the above-described device may be recorded on a computer-readable recording medium and loaded into a computer system for execution. Here, the device may be, for example, the information processing terminal 10, the detection unit 20, or the information processing device 30. Note that the term "computer system" herein includes hardware such as an operating system (OS) and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and compact disks (CDs)-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0120] The above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line. The program may also be a program for realizing some of the functions described above, or may be a so-called differential file or differential program that can realize the functions described above in combination with a program already recorded in the computer system. [Explanation of symbols]

[0121] 1...information processing system, 10...information processing terminal, 11...first processor, 12...first memory unit, 13...input receiving unit, 14...first communication unit, 15...display unit, 16...first control unit, 20...detection unit, 30...information processing device, 31...second processor, 32...second memory unit, 34...second communication unit, 36...second control unit, 161...imaging control unit, 162...first processing unit, 163...display control unit, 361...second processing unit, C...imaging unit, UR...subject

Claims

1. A control method for an information processing system including an information processing terminal and an information processing device, a subject model based on a moving image of a series of actions by the subject, and a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating a trajectory of a tool used by the subject in the series of actions, are displayed on a display unit; the subject model is a three-dimensional model representing the series of actions performed by the subject; the tool model is a three-dimensional model that shows the trajectory of the tool during the series of movements performed by the subject; Control method.

2. The control method includes: acquiring the moving image, and generating the subject model based on the acquired moving image; acquiring the trajectory information, and generating the tool model based on the acquired trajectory information; The control method according to claim 1 .

3. calculating two distances: a first distance between the three-dimensional coordinates of the tool model in a first reference orientation among a plurality of reference orientations serving as references for the series of movements and the three-dimensional coordinates of the tool model in a second reference orientation among the plurality of reference orientations that is different from the first reference orientation; and a second distance between the three-dimensional coordinates of the tool model in the second reference orientation and the three-dimensional coordinates of the tool model in a third reference orientation among the plurality of reference orientations that is different from both the first reference orientation and the second reference orientation; setting three-dimensional coordinates indicating positions of the hands of the subject model in the second reference posture based on the three-dimensional coordinates indicating positions of the hands of the subject model in the first reference posture and the first distance; setting three-dimensional coordinates indicating positions of the hands of the subject model in the third reference posture based on the three-dimensional coordinates indicating positions of the hands of the subject model in the first reference posture and the second distance; synchronizing a three-dimensional coordinate system of the subject model with a three-dimensional coordinate system of the tool model based on three coordinates: a three-dimensional coordinate indicating a position of the hand of the subject model in the first reference posture, a three-dimensional coordinate indicating a position of the hand of the subject model in the second reference posture, and a three-dimensional coordinate indicating a position of the hand of the subject model in the third reference posture; The control method according to claim 2 .

4. calculating a center of gravity of the three coordinates, and calculating a rotation matrix for rotating a three-dimensional coordinate system of the subject model based on the calculated center of gravity; synchronizing the three-dimensional coordinate system of the subject model with the three-dimensional coordinate system of the tool model based on the calculated rotation matrix; The control method according to claim 3 .

5. identifying, from a time series of three-dimensional coordinates indicating the hand position of the subject model, three-dimensional coordinates whose distance from the three-dimensional coordinates indicating the hand position of the subject model in the first reference posture is closest to the first distance, and identifying the identified three-dimensional coordinates as first three-dimensional coordinates indicating the hand position of the subject model in the second reference posture; identifying, from a time series of three-dimensional coordinates indicating the hand position of the subject model, three-dimensional coordinates whose distance from the three-dimensional coordinates indicating the hand position of the subject model in the first reference posture is closest to the second distance, and identifying the identified three-dimensional coordinates as second three-dimensional coordinates indicating the hand position of the subject model in the third reference posture; synchronizing a three-dimensional coordinate system of the subject model with a three-dimensional coordinate system of the tool model based on the identified first three-dimensional coordinates and the identified second three-dimensional coordinates; The control method according to claim 4.

6. determining whether or not the posture of the subject has reached the first reference posture based on the acquired moving image, and starting acquisition of the trajectory information when it is determined that the posture of the subject has reached the first reference posture; The control method according to claim 5 .

7. The detection unit detects the angular velocity and acceleration of the tool, the control method includes resetting the detection unit when it is determined that the posture of the subject has become the first reference posture. The control method according to claim 6.

8. the series of movements is a golf movement, the first reference position is an address position, the second reference position is a halfway back position, the third reference position is a top position; The control method according to claim 7.

9. changing orientations of the subject model and the tool model displayed on the display unit in accordance with the received operation; The control method according to claim 1 .

10. A control method for controlling an information processing device, comprising: a subject model based on a moving image of a series of actions by the subject, and a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating a trajectory of a tool used by the subject in the series of actions, are displayed on a display unit; the subject model is a three-dimensional model representing the series of actions performed by the subject; the tool model is a three-dimensional model that shows the trajectory of the tool during the series of movements performed by the subject; Control method.

11. A computer is caused to implement the control method according to claim 10. program.

12. An information processing system including an information processing terminal and an information processing device, a display control unit that causes a display unit to display, together with a subject model based on moving images of a series of actions by the subject, a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating a trajectory of a tool used by the subject in the series of actions, the subject model is a three-dimensional model representing the series of actions performed by the subject; the tool model is a three-dimensional model that shows the trajectory of the tool during the series of movements performed by the subject; Information processing system.

13. a display control unit that causes a display unit to display, together with a subject model based on moving images of a series of actions by the subject, a tool model based on trajectory information detected by a detection unit that detects trajectory information indicating a trajectory of a tool used by the subject in the series of actions, the subject model is a three-dimensional model representing the series of actions performed by the subject; the tool model is a three-dimensional model that shows the trajectory of the tool during the series of movements performed by the subject; Information processing device.

Images