Information processing method, program, and information processing device

Abstract

To provide an information processing method and device and a program that easily acquire data expressing the motion of persons in three dimensions.SOLUTION: An information processing device 300 in an information processing system comprises: a moving-image data acquisition unit 3001 for receiving from a user terminal 200; a two-dimensional motion data acquisition unit 3002 for acquiring observation data expressing the motion in two dimensions of a person who acquires data of a moving-image expressing what a user's motion is like; a simulation data acquisition unit 3004 for acquiring numerical simulation data simulating the motion of a person in three dimensions; and a three-dimensional motion data estimation unit 3005 for estimating data expressing in three dimensions the motion of a person corresponding to the observation data acquired by the two-dimensional motion data acquisition unit, on the basis of the observation data acquired from a camera 100 by the two-dimensional motion data acquisition unit of the user terminal and numerical simulation data acquired by the simulation data acquisition unit 3004.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to an information processing method and the like.

Background Art

[0002] Conventionally, a technique for three-dimensionally measuring human movements has been known.

[0003] Also, a technique for estimating data representing human movements in three dimensions from a two-dimensional image is known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in order to perform three-dimensional measurement of human movements, it may be necessary to use a plurality of measurement devices such as cameras, or to attach markers to the target person or the tools used by the person. Also, in Patent Document 1, in order to utilize supervised learning, a large amount of teacher data may be required to generate a model for estimating data representing human movements in three dimensions from a two-dimensional image.

[0006] Therefore, in view of the above problems, an object is to provide a technique capable of easily acquiring data representing human movements in three dimensions.

Means for Solving the Problems

[0007] To achieve the above object, in one embodiment of the present disclosure, an information processing apparatus includes a first acquisition step of acquiring observation data representing a human movement in two dimensions, A second acquisition step in which an information processing apparatus acquires data of a numerical simulation that simulates human motion in three dimensions, Based on the observation data acquired in the first acquisition step and the data of the numerical simulation acquired in the second acquisition step, an estimation step of estimating data representing human motion in three dimensions corresponding to the observation data acquired in the first acquisition step by using data assimilation processing, An information processing method is provided.

[0008] In another embodiment of the present disclosure, In an information processing apparatus, A first acquisition step of acquiring observation data representing human motion in two dimensions, A second acquisition step of acquiring data of a numerical simulation that simulates human motion in three dimensions, Based on the observation data acquired in the first acquisition step and the data of the numerical simulation acquired in the second acquisition step, an estimation step of estimating data representing human motion in three dimensions corresponding to the observation data acquired in the first acquisition step by using data assimilation processing, A program is provided.

[0009] In still another embodiment of the present disclosure, A first acquisition unit that acquires observation data representing human motion in two dimensions, A second acquisition unit that acquires data of a numerical simulation that simulates human motion in three dimensions, Based on the observation data acquired by the first acquisition unit and the data of the numerical simulation acquired by the second acquisition unit, an estimation unit that estimates data representing human motion in three dimensions corresponding to the observation data acquired in the first acquisition step by using data assimilation processing, An information processing apparatus is provided.

Advantages of the Invention

[0010] According to the above-described embodiment, data representing a person's movement in three dimensions can be easily acquired.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described with reference to the drawings.

[0013] [Overview of the Information Processing System] Referring to FIGS. 1 to 3, the overview of the information processing system 1 according to this embodiment will be described.

[0014] FIG. 1 is a diagram showing an example of the information processing system 1. FIG. 2 is a diagram showing an example of a moving image (moving image 20) representing the state of a user's operation. FIG. 3 is a diagram showing another example of a moving image (moving image 30) representing the state of a user's operation. Specifically, FIGS. 2 and 3 are specific examples of moving images representing the state of a golf swing operation by a user.

[0015] In FIG. 2, frames 21 to 28 out of all the frames of the moving image 20 are excerpted and shown. Frame 21 represents the address state in the golf swing operation by the user. Frame 22 represents the take-back state in the golf swing operation by the user. Frame 23 represents the backswing state in the golf swing operation by the user. Frame 24 represents the top state in the golf swing operation by the user. Frame 25 represents the halfway down state in the golf swing operation by the user. Frame 26 represents the impact state in the golf swing operation by the user. Frame 27 represents the follow state in the golf swing operation by the user. Frame 28 represents the finish state in the golf swing operation by the user. Similarly, in FIG. 3, frame 31 out of all the frames of the moving image 30 is excerpted and shown. Frame 31 represents the halfway down state in the golf swing operation by the user.

[0016] As shown in FIG. 1, the information processing system 1 includes a camera 100, a user terminal 200, and an information processing device 300.

[0017] The information processing system 1 estimates, in the information processing apparatus 300, data representing the user's motion in three dimensions (hereinafter, "three-dimensional motion data") based on moving image data obtained by the camera 100 and containing images of the user's motion. Then, the information processing system 1 performs an analysis regarding the user's motion (hereinafter, "motion analysis") based on the three-dimensional motion data, and notifies the user of the analysis result.

[0018] For example, the information processing system 1 acquires three-dimensional motion data corresponding to a specific motion of the user based on moving image data in which the specific motion of the user is captured. The specific motion is, for example, a golf swing motion or a putting motion (hereinafter, "golf swing motion, etc."). Thereby, the information processing system 1 can analyze the user's swing motion, etc. based on the three-dimensional motion data regarding the golf swing motion, etc. Therefore, the information processing system 1 can perform a diagnosis regarding the user's swing motion, etc. (hereinafter, "swing diagnosis"), or select and propose a recommended club, etc. suitable for the user's swing motion, etc. Further, the specific motion may be, for example, a baseball batting motion, a baseball catching motion, a tennis serve motion, or a tennis swing motion such as a forehand motion. Further, the specific motion may be, for example, a motion without using a tool, such as the user's walking motion or running motion. Hereinafter, in the present embodiment, the case where the information processing system 1 estimates the three-dimensional motion regarding the golf swing motion will be mainly described.

[0019] The camera 100 captures the user's motion and acquires a moving image representing the state of the motion. The moving image is composed of a series of still images (hereinafter, "frames").

[0020] The camera 100 is, for example, a so-called two-dimensional camera and acquires a two-dimensional moving image. Further, the camera 100 may be a three-dimensional camera capable of acquiring information in the depth direction of the two-dimensional moving image in addition to the two-dimensional moving image. The information in the depth direction of the two-dimensional moving image is, for example, information representing the position in the depth direction of the object shown for each pixel of each frame constituting the two-dimensional moving image.

[0021] The camera 100 acquires a moving image representing the state of the user's movement in response to an operation by a photographer different from the user who performs the operation, for example. Further, the camera 100 may acquire a moving image representing the state of the user's movement in response to the user's operation by a self-timer function or the like.

[0022] The camera 100 may acquire a moving image representing the user's movement captured from one viewpoint, or may acquire a moving image representing the user's movement captured from a plurality of viewpoints. In the latter case, for example, moving images representing the user's movement viewed from different viewpoints at the same timing are acquired by a plurality of cameras 100. Further, by changing the position of one camera 100, images representing the user's movement with different viewpoints may be sequentially acquired.

[0023] For example, as shown in FIG. 2, the camera 100 images the golf swing motion from the front of the user in a form facing the user. Further, as shown in FIG. 3, the camera 100 may image the golf swing motion from behind the user. The rear of the user means the rear of the user when the direction of the virtual fly ball line (hereinafter, "virtual fly ball line") assumed by the user is defined as "front". Further, the camera 100 may acquire both a moving image representing the golf swing motion from the front of the user and a moving image of the golf swing motion from behind the user. That is, the camera 100 may acquire a moving image representing the user's movement viewed from one viewpoint, or may acquire a moving image representing the user's movement viewed from a plurality of viewpoints.

[0024] In FIG. 1, the camera 100 and the user terminal 200 are drawn separately, but the camera 100 may be built in the user terminal 200, or may be provided separately from the user terminal 200. In the latter case, the output of the camera 100 (that is, the data of the moving image) may be taken into the user terminal 200 by communication through the communication interface 206 described later, or may be taken into the user terminal 200 through the recording medium 201A described later.

[0025] The user terminal 200 is a terminal device used by the user. The user terminal 200 may be, for example, a terminal device arranged in a golf lesson facility, a shop, etc., or may be a terminal device possessed by the user.

[0026] The user terminal 200 is, for example, a portable terminal device, that is, a mobile terminal. The mobile terminal is, for example, a smartphone, a tablet terminal, a laptop PC (Personal Computer), etc. Also, the user terminal 200 may be a stationary terminal device. The stationary terminal device is, for example, a desktop PC.

[0027] The user terminal 200 is communicably connected to the information processing device 300 through a predetermined communication line. The predetermined communication line includes, for example, a local area network (LAN: Local Area Network). Also, the predetermined communication line may include a wide area network (WAN: Wide Area Network). The wide area network includes, for example, the Internet. Also, the wide area network may include a mobile communication network with a base station at the end or a satellite communication network using communication satellites. Also, the predetermined communication line may include, for example, a short-distance communication line using a predetermined wireless communication standard such as WiFi, Bluetooth (registered trademark), local 5G (5 th Generation).

[0028] The user terminal 200 captures a moving image representing the state of the user's operation from the camera 100 and transmits it to the information processing device 300. Then, the user terminal 200 presents to the user, through the display device 208 described later, the information regarding the result of the motion analysis returned from the information processing device 300. The information regarding the result of the motion analysis is, for example, when targeting a golf swing motion, etc., information representing the diagnosis result of the golf swing based on the motion analysis. Also, the information regarding the result of the motion analysis may be, when targeting a golf swing motion, etc., information representing recommended equipment such as recommended golf clubs and balls based on the motion analysis.

[0029] The information processing apparatus 300 acquires three-dimensional motion data of the user based on a moving image received from the user terminal 200 and representing the state of the user's motion. Then, the information processing apparatus 300 performs motion analysis of the user's motion based on the acquired three-dimensional motion data and returns information regarding the result of the motion analysis to the user terminal 200.

[0030] The information processing apparatus 300 is, for example, a server apparatus with relatively high processing power. The server apparatus may be a cloud server, an on-premises server, or an edge server. Also, depending on the required processing power, the information processing apparatus may be a terminal device with lower processing power than the server apparatus. The terminal device may be a stationary terminal device or a portable terminal device (mobile terminal).

[0031] [Configuration of the information processing system] Next, in addition to FIG. 1, referring to FIGS. 4 and 5, the configuration of the information processing system 1 will be described.

[0032] [Configuration of the user terminal] FIG. 4 is a block diagram showing an example of the configuration of the user terminal 200.

[0033] The functions of the user terminal 200 may be realized by any hardware, or any combination of hardware and software, etc. For example, as shown in FIG. 4, the user terminal 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a communication interface 206, an input device 207, a display device 208, and an audio output device 209. These components are connected by a bus B2. Also, as described above, when the camera 100 is built into the user terminal 200, the camera 100 may be connected to the bus B2 in the same manner as the other components.

[0034] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. The recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB (Universal Serial Bus) memory, and the like. Thereby, the user terminal 200 can read various data used in processing through the recording medium 201A, store them in the auxiliary storage device 202, or install a program for realizing various functions.

[0035] Note that the user terminal 200 may acquire various data and programs used in processing from an external device (for example, the information processing device 300) through the communication interface 206.

[0036] The auxiliary storage device 202 stores installed various programs and also stores files, data, etc. necessary for various processes. The auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), an SSD (Solid State Disc), a flash memory, and the like.

[0037] When there is an instruction to start a program, the memory device 203 reads the program from the auxiliary storage device 202 and stores it. The memory device 203 includes, for example, a DRAM (Dynamic Random Access Memory) and an SRAM (Static Random Access Memory).

[0038] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203 and realizes various functions related to the user terminal 200 according to the programs.

[0039] The communication interface 206 is used as an interface for communicably connecting to an external device. Thereby, the user terminal 200 can acquire moving image data from the camera 100 through the communication interface 206. Further, the user terminal 200 can communicate with an external device such as the information processing device 300 through the communication interface 206. Also, the communication interface 206 may have a plurality of types of communication interfaces according to the communication method or the like with the connected device.

[0040] The input device 207 receives various inputs from the user.

[0041] The input device 207 includes, for example, an input device in a form that receives a mechanical input from the user (hereinafter, "mechanical input device"). The mechanical input device includes, for example, buttons, toggles, levers, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, a keyboard, a mouse, and the like.

[0042] Also, the input device 207 may include a voice input device capable of receiving a voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0043] Also, the input device 207 may include a gesture input device capable of receiving a gesture input from the user. The gesture input device includes, for example, a camera capable of imaging the state of the user's gesture.

[0044] Also, the input device 207 may include a biometric input device capable of receiving a biometric input from the user. The biometric input device includes, for example, a camera capable of acquiring image data including information about the user's fingerprint or iris.

[0045] The display device 208 displays an information screen, an operation screen, etc. toward the user. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

[0046] The sound output device 209 conveys various types of information to the user of the user terminal 200 by sound. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[0047] <Configuration of Information Processing Device> FIG. 5 is a block diagram showing an example of the configuration of the information processing device 300.

[0048] The functions of the information processing device 300 may be realized by any hardware, or any combination of hardware and software, etc. For example, as shown in FIG. 5, the information processing device 300 includes an external interface 301, an auxiliary storage device 302, a memory device 303, a CPU 304, a high-speed arithmetic device 305, a communication interface 306, an input device 307, a display device 308, and a sound output device 309. These components are connected by a bus B3.

[0049] The external interface 301 functions as an interface for reading data from the recording medium 301A and writing data to the recording medium 301A. The recording medium 301A includes, for example, a flexible disk, a CD, a DVD, a BD, an SD memory card, a USB memory, and the like. Thereby, the information processing device 300 can read various data used in processing through the recording medium 301A and store it in the auxiliary storage device 302, or install a program for realizing various functions.

[0050] Note that the information processing device 300 may acquire various data and programs used in processing from an external device through the communication interface 306.

[0051] The auxiliary storage device 302 stores installed various programs and also stores files, data, etc. necessary for various processes. The auxiliary storage device 302 includes, for example, an HDD, an SSD, a flash memory, and the like.

[0052] When there is an instruction to start a program, the memory device 303 reads and stores the program from the auxiliary storage device 302. The memory device 303 includes, for example, DRAM and SRAM.

[0053] The CPU 304 executes various programs loaded from the auxiliary storage device 302 into the memory device 303, and realizes various functions related to the information processing device 300 according to the programs.

[0054] The high-speed arithmetic unit 305 operates in conjunction with the CPU 304 and performs arithmetic processing at a relatively high speed compared to the CPU 304. The high-speed arithmetic unit 305 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), and the like.

[0055] Note that the high-speed arithmetic unit 305 may be omitted depending on the speed of the required arithmetic processing.

[0056] The communication interface 306 is used as an interface for communicably connecting to an external device. Thereby, the information processing device 300 can communicate with an external device such as the user terminal 200 through the communication interface 306. Further, the communication interface 306 may have a plurality of types of communication interfaces according to the communication method and the like with the connected device.

[0057] The input device 307 receives various inputs from the user.

[0058] The input device 307 includes, for example, a mechanical input device that receives a mechanical operation input from the user. The mechanical input device includes, for example, buttons, toggles, levers, a touch panel mounted on the display device 308, a touch pad provided separately from the display device 308, a keyboard, a mouse, and the like.

[0059] In addition, the input device 307 includes, for example, a voice input device capable of receiving voice input from a user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0060] In addition, the input device 307 includes, for example, a gesture input device capable of receiving gesture input from a user. The gesture input device includes, for example, a camera capable of imaging the state of the user's gestures.

[0061] In addition, the input device 307 includes, for example, a biometric input device capable of receiving biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about the user's fingerprint or iris.

[0062] The display device 308 displays an information screen or an operation screen for the user. The display device 308 is, for example, a liquid crystal display, an organic EL display, or the like.

[0063] The sound output device 309 conveys various information to the user of the information processing device 300 by sound. The sound output device 309 is, for example, a buzzer, an alarm, a speaker, or the like.

[0064] [First Example of the Functional Configuration of the Information Processing System] Next, with reference to FIGS. 6 to 9, a first example of the functional configuration of the information processing system 1 will be described.

[0065] FIG. 6 is a functional block diagram showing a first example of the functional configuration of the information processing system. FIG. 7 is a diagram for explaining an example of two-dimensional motion data. Specifically, FIG. 7 is a diagram showing feature points representing the user's motion for each of a plurality of frames included in a moving image representing the state of a golf swing motion by the user. FIG. 8 is a diagram for explaining an example of a three-dimensional numerical simulation of the user's motion. Specifically, FIG. 8 is a diagram for explaining an example of a three-dimensional numerical simulation related to the golf swing motion by the user. FIG. 9 is a diagram schematically explaining an example of data assimilation processing.

[0066] Further, in FIG. 7, data 70 of feature points (black circles in the figure) representing the user's actions for each of a plurality of frames of a moving image representing the user's golf swing action is schematically visualized and presented. In FIG. 7, data 71 to 78 of feature points for each frame of a part (frames 21 to 28) of all the frames of the moving image 20 in FIG. 2 are excerpted and shown. Data 71 represents the data of the feature points corresponding to frame 21 in the address state in the golf swing action by the user. Data 72 represents the data of the feature points corresponding to frame 22 in the take-back state in the golf swing action by the user. Data 73 represents the data of the feature points corresponding to frame 23 in the backswing state in the golf swing action by the user. Data 74 represents the data of the feature points corresponding to frame 24 in the top state in the golf swing action by the user. Data 75 represents the data of the feature points corresponding to frame 25 in the halfway down state in the golf swing action by the user. Data 76 represents the data of the feature points corresponding to frame 26 in the impact state in the golf swing action by the user. Data 77 represents the data of the feature points corresponding to frame 27 in the follow state in the golf swing action by the user. Data 78 represents the data of the feature points corresponding to frame 28 in the finish state in the golf swing action by the user.

[0067] As shown in FIG. 6, the user terminal 200 includes an application screen display processing unit 2001, a moving image data acquisition unit 2002, a moving image data transmission unit 2003, and an analysis result data acquisition unit 2004. These functions are realized, for example, when an application program (hereinafter simply referred to as "application") installed in the auxiliary storage device 202 is loaded into the memory device 203 and executed by the CPU 204.

[0068] The application screen display processing unit 2001 causes the display device 208 to display a screen related to the application (hereinafter referred to as "application screen").

[0069] The moving image data acquisition unit 2002 acquires data of a moving image representing the state of the user's operation from the camera 100. For example, the moving image data acquisition unit 2002 acquires data of a moving image representing the state of the user's operation from the camera 100 in response to an input from the user on a predetermined application screen using the input device 207. At this time, the moving image data acquisition unit 2002 may acquire a moving image that has already been captured by the camera 100, or may acquire the latest moving image acquired by the camera 100 in real time. The data of the moving image includes data of a two-dimensional moving image. Further, the data of the moving image may include information in the depth direction of the two-dimensional moving image in addition to the data of the two-dimensional moving image.

[0070] The moving image data transmission unit 2003 transmits the data of the moving image representing the state of the user's operation, which is acquired by the moving image data acquisition unit 2002, to the information processing apparatus 300 through the communication interface 206. For example, the moving image data transmission unit 2003 transmits the moving image data acquired from the camera 100 to the information processing apparatus 300 in response to an input from the user on a predetermined application screen using the input device 207.

[0071] The analysis result data acquisition unit 2004 acquires reply data including information regarding the result of the motion analysis, which is received from the information processing apparatus 300. The content of the reply data including the information representing the result of the motion analysis is displayed on the display device 208 by the application screen display processing unit 2001. Thereby, the user can confirm the content of the reply data including the information regarding the result of the motion analysis.

[0072] As shown in FIG. 6, the information processing apparatus 300 includes a moving image data acquisition unit 3001, a two-dimensional motion data acquisition unit 3002, a simulator unit 3003, a simulation data acquisition unit 3004, a three-dimensional motion data estimation unit 3005, a motion analysis unit 3006, and an analysis result transmission unit 3007. These functions are realized, for example, by a program installed in the auxiliary storage device 302 being loaded into the memory device 303 and executed by the CPU 304.

[0073] The moving image data acquisition unit 3001 acquires data of a moving image representing the state of a user's actions received from the user terminal 200.

[0074] Based on the data of the moving image (specifically, data of a two-dimensional moving image) acquired by the moving image data acquisition unit 3001, the two-dimensional motion data acquisition unit 3002 acquires observation data (hereinafter, "two-dimensional motion data") representing the actions of the person shown in the moving image in two dimensions. The two-dimensional motion data is a time series collection of data representing the motion state of a person at each time point corresponding to each frame of the moving image data in two dimensions. The data representing the motion state of a person at each time point in two dimensions includes, for example, data representing the position of a person's body parts or the parts of a tool held by the person in two dimensions and data representing the posture angle in two dimensions.

[0075] Also, when the moving image data acquisition unit 3001 has acquired data of moving images viewed from a plurality of different viewpoints, the two-dimensional motion data acquisition unit 3002 acquires two-dimensional motion data for each of the plurality of moving images corresponding to the plurality of viewpoints.

[0076] For example, as shown in FIG. 7, the two-dimensional motion data acquisition unit 3002 extracts a plurality of feature points related to the actions of the subject user for each frame of the moving image.

[0077] The plurality of feature points include feature points (hereinafter, "physical feature points") representing the body parts of the subject user on the image of the frame. The physical feature points represent the joint positions in the user's skeleton, and the physical feature points included in the plurality of feature points include points on the image corresponding to the head, shoulders, elbows, wrists, waist, knees, ankles, etc. Thereby, the two-dimensional motion data acquisition unit 3002 can acquire two-dimensional motion data representing the position and posture angle of the user's body parts in two dimensions.

[0078] In addition, among the plurality of feature points, there are feature points representing the parts of the tool held by the user. For example, as shown in FIG. 7, feature points corresponding to the head of the club held by the user are extracted. Thereby, the two-dimensional motion data acquisition unit 3002 can acquire two-dimensional motion data representing the position and posture angle of the part of the tool held by the user in two dimensions.

[0079] In addition, the two-dimensional motion data acquisition unit 3002 adjusts the coordinate system and the scale of the coordinate system of the two-dimensional motion data so as to be comparable with the two-dimensional simulation data described later. For example, the two-dimensional motion data acquisition unit 3002 acquires two-dimensional motion data on the premise of a coordinate system along the scale of the real world, with the subject person (i.e., the user) as a reference.

[0080] The simulator unit 3003 performs a numerical simulation that simulates human motion in three dimensions using a three-dimensional body model.

[0081] For example, as shown in FIG. 8, the simulator unit 3003 performs a numerical simulation by simulating a golf swing motion or the like of a person using the body model MD1 on a three-dimensional XYZ orthogonal coordinate system.

[0082] The simulation data acquisition unit 3004 acquires time-series output data of a numerical simulation that simulates human motion in three dimensions using the simulator unit 3003 (hereinafter, "three-dimensional simulation data"). The three-dimensional simulation data is a collection of time-series data representing the human motion state in three dimensions at each discretized time point.

[0083] In addition, the simulation data acquisition unit 3004 acquires data (hereinafter, "two-dimensional simulation data") viewed from the same viewpoint as the two-dimensional motion data acquired by the two-dimensional motion data acquisition unit 3002 based on the acquired three-dimensional simulation data. In the two-dimensional simulation, similar to the three-dimensional simulation data, it is a collection of time-series data representing the human motion state in two dimensions at each discretized time point.

[0084] At this time, the simulation data acquisition unit 3004 adjusts the dimensions of each part of the body model according to the physical dimensions (e.g., height) of the user, and causes the simulator unit 3003 to perform a numerical simulation using the adjusted body model. For example, the simulation data acquisition unit 3004 adjusts the dimensions of each part of the body model based on the input content of the user's height at the user terminal 200. Further, the simulation data acquisition unit 3004 may measure the dimensions representing the physical characteristics such as the user's height based on the two-dimensional motion data, and adjust the dimensions of each part of the body model based on the measurement result.

[0085] Also, when the two-dimensional motion data acquisition unit 3002 acquires the two-dimensional motion data for each of the plurality of moving image data corresponding to the plurality of viewpoints, the simulation data acquisition unit 3004 acquires the two-dimensional simulation data viewed from each of the same plurality of viewpoints.

[0086] For example, as shown in FIG. 8, by projecting the three-dimensional simulation data onto the plane PL1 as the YZ plane, the two-dimensional simulation data when the body model MD1 is viewed from the front can be acquired. Thereby, for example, the two-dimensional motion data when the user is viewed from the front obtained from the data of the moving image 20 in FIG. 2 and the two-dimensional simulation data viewed from the same viewpoint can be acquired.

[0087] Also, the simulation data acquisition unit 3004 may acquire three-dimensional simulation data using the simulator unit 3003 based on the reference data representing a specific human motion in three dimensions. Thereby, the simulator unit 3003 can perform a numerical simulation simulating a specific human motion within the frame of the specific motion. Therefore, it is possible to suppress a situation in which three-dimensional motion data far from the specific motion is estimated based on the three-dimensional simulation data.

[0088] Hereinafter, three-dimensional simulation data and two-dimensional simulation data are collectively referred to as "simulation data".

[0089] Based on the two-dimensional motion data and the simulation data, the three-dimensional motion data estimation unit 3005 estimates data (hereinafter referred to as "three-dimensional motion data") that represents a person's motion corresponding to the two-dimensional motion data in three dimensions. Specifically, the three-dimensional motion data estimation unit 3005 estimates the three-dimensional motion data by using data assimilation processing based on the two-dimensional motion data and the simulation data.

[0090] The data assimilation process is, for example, a sequential data assimilation process. Specifically, the three-dimensional motion data estimation unit 3005 estimates the three-dimensional motion data by reflecting the two-dimensional motion data through filtering by a Bayesian filter based on Bayes' theorem, using the simulation data as a predicted value. For example, the three-dimensional motion data estimation unit 3005 performs a sequential data assimilation process using an extended Kalman filter (EKF) as a Bayesian filter. Also, the three-dimensional motion data estimation unit 3005 may perform a sequential data assimilation process using an unscented Kalman filter (UKF), an ensemble Kalman filter (EnKF), a particle filter, or the like as a Bayesian filter.

[0091] For example, as shown in FIG. 9, the simulation data acquisition unit 3004 executes a numerical simulation using the simulator unit 3003 for the parameter 901 that represents a golf swing motion in three dimensions, and acquires three-dimensional simulation data. Then, the simulation data acquisition unit 3004 acquires two-dimensional simulation data 902 based on the three-dimensional simulation data representing the golf motion.

[0092] The three-dimensional motion data estimation unit 3005 estimates three-dimensional motion data using an extended Kalman filter based on the two-dimensional simulation data 902 and the two-dimensional motion data 903. Specifically, the three-dimensional motion data can be estimated by updating the three-dimensional simulation data using the matrix product of the difference between the two-dimensional simulation data 902 and the two-dimensional motion data and the Kalman gain.

[0093] Also, the data assimilation process may be a non-sequential data assimilation process.

[0094] Also, the three-dimensional motion data estimation unit 3005 may determine the dimensions of a predetermined part of the body model used in the numerical simulation of the simulator unit 3003 by the data assimilation process. The dimensions of a predetermined part of the body model are, for example, the dimensions of the rigid links connecting the joints of the body model. For example, as variable parameters, in addition to the data representing human motion (see, for example, parameter 901 in FIG. 9), the dimensions of the rigid links of the body model are adopted. Thereby, the three-dimensional motion data estimation unit 3005 can estimate the dimensions of the rigid links of the body model by the data assimilation process. Therefore, for example, the three-dimensional motion data estimation unit 3005 can determine the dimensions of the rigid links by calculating the average value of the estimated values of the dimensions of the rigid links obtained for each time series of the two-dimensional motion data. The simulation data acquisition unit 3004 can reflect the determined dimensions of the rigid links as definite values in the body model and acquire simulation data again using the simulator unit 3003. Then, the three-dimensional motion data estimation unit 3005 estimates the three-dimensional motion data again using the data assimilation process without including the dimensions of the rigid links in the variable parameters based on the simulation data and the two-dimensional motion data. Thereby, the accuracy of the three-dimensional motion data can be improved.

[0095] As described above, when it is possible to acquire moving image data for each of a plurality of viewpoints, it is also possible to acquire three-dimensional motion data based on the same principle as a stereo camera. However, since the moving image data acquired by the camera 100 includes the influence of lens distortion, the accuracy of the position of the feature points acquired from the moving image data may be relatively low. As a result, in the three-dimensional motion data acquired based on the same principle as a stereo camera, the length of the user's arm or the club may extend or contract at each time.

[0096] On the other hand, in this example, since the length of the golf club and the dimensions of the rigid links are fixed and numerical simulation is performed, such a problem cannot occur in the three-dimensional motion data estimated based on the three-dimensional simulation data. Further, in this example, instead of assuming the two-dimensional motion data itself based on the moving image data, a form is adopted in which the two-dimensional motion data is reflected in the three-dimensional simulation data using data assimilation processing. Therefore, in this example, the information processing apparatus 300 can improve the accuracy of the three-dimensional motion data while considering the influence of lens distortion that may be included in the two-dimensional motion data.

[0097] The motion analysis unit 3006 performs motion analysis of a person corresponding to the moving image data based on the three-dimensional motion data as the estimation result of the three-dimensional motion data estimation unit 3005. As a result, the motion analysis unit 3006 can perform a more detailed motion analysis than when performing motion analysis based on two-dimensional motion data, for example. For example, the motion analysis unit 3006 analyzes a person's motion based on the three-dimensional motion data and outputs information on advice and improvement points regarding the person's motion (for example, information on the diagnosis result of swing diagnosis). As a result, the motion analysis unit 3006 can output information on more appropriate and more accurate advice and improvement points based on a more detailed motion analysis based on the three-dimensional motion data. Further, the motion analysis unit 3006 may analyze the motion of a person corresponding to the moving image data based on the three-dimensional motion data and output information on a tool (for example, a golf club or the like) suitable for the person's motion. As a result, the motion analysis unit 3006 can more appropriately select a tool suitable for a person's motion based on a more detailed motion analysis based on the three-dimensional motion data.

[0098] The analysis result transmission unit 3007 transmits information regarding the result of the motion analysis by the motion analysis unit 3006 to the user terminal 200 through the communication interface 306. As a result, the user can confirm information regarding the result of the motion analysis through the application screen of the user terminal 200.

[0099] As described above, in this example, the information processing apparatus 300 can estimate three-dimensional motion data representing the motion of a person corresponding to the two-dimensional motion data by using the data assimilation process based on the two-dimensional motion data and the simulation data.

[0100] [First Example of Operation of Information Processing System] Next, with reference to FIG. 10, a first example of the operation of the information processing system 1 will be described.

[0101] FIG. 10 is a sequence diagram schematically showing a first example of the operation of the information processing system 1.

[0102] As shown in FIG. 10, the user terminal 200 activates an application in response to a predetermined input from the user using the input device 207 (step S102).

[0103] After the completion of the process in step S102, the moving image data acquisition unit 2002 acquires moving image data representing the state of the user's actions from the camera 100 in response to a predetermined input from the user on a predetermined application screen using the input device 207 (step S104).

[0104] After the completion of the process in step S104, the moving image data transmission unit 2003 transmits the moving image data to the information processing device 300 through the communication interface 206 (step S106).

[0105] The moving image data acquisition unit 3001 acquires the moving image data transmitted (uploaded) from the user terminal 200 in the process of step S106 (step S108).

[0106] After the completion of the process in step S108, the two-dimensional motion data acquisition unit 3002 acquires two-dimensional motion data by extracting feature points representing the user's actions in each frame of the moving image data (step S110).

[0107] After the completion of the process in step S110, the simulation data acquisition unit 3004 acquires three-dimensional simulation data using the simulator unit 3003 (step S112).

[0108] After the completion of the process in step S112, the simulation data acquisition unit 3004 acquires two-dimensional simulation data based on the three-dimensional simulation data (step S114).

[0109] After the completion of the process in step S114, the three-dimensional motion data estimation unit 3005 estimates three-dimensional motion data representing the actions of a person corresponding to the two-dimensional motion data using the data assimilation process based on the data acquired in steps S110 and S114 (step S116).

[0110] After the completion of the process in step S116, based on the three-dimensional motion data acquired in the process of step S116, the motion analysis unit 3006 performs motion analysis on the human motion corresponding to the two-dimensional motion data acquired in step S110 (step S118).

[0111] After the completion of the process in step S118, the analysis result transmission unit 3007 transmits, through the communication interface 306, reply data including information regarding the result of the motion analysis to the user terminal 200 (step S120).

[0112] The analysis result data acquisition unit 2004 acquires the reply data transmitted from the information processing apparatus 300 in the process of step S120 (step S122).

[0113] After the completion of the process in step S122, the application screen display processing unit 2001 causes the display device 208 to display the information regarding the result of the motion analysis included in the reply data (step S124).

[0114] [Second Example of the Functional Configuration of the Information Processing System] Next, with reference to FIG. 11, a second example of the functional configuration of the information processing system 1 will be described.

[0115] In this example, the same or corresponding components as those in the above-described first example (FIG. 6) are denoted by the same reference numerals, and the description will be centered on the parts different from the above-described first example.

[0116] In this example, the information processing apparatus 300 is different from the above-described first example in that it includes an event timing setting unit 3008.

[0117] The event timing setting unit 3008 sets the timings of a plurality of events in a specific operation of the user for the time-series data included in the two-dimensional motion data. Specifically, the event timing setting unit 3008 extracts data corresponding to each of the plurality of events from the time-series data included in the two-dimensional motion data, and performs a setting to associate each data with an event. An event in a specific operation of the user is, for example, an event representing a characteristic motion state in a specific operation of the user. An event in a specific operation of the user is, for example, when the specific operation is a golf swing operation, address, takeback, backswing, top, halfway down, impact, follow-through, and finish, etc.

[0118] The event timing setting unit 3008 sets the timings of a plurality of events for the time-series data included in the two-dimensional motion data, for example, in response to a manual setting input from the user or an operator of the information processing apparatus 300.

[0119] Further, the event timing setting unit 3008 may automatically set the timings of a plurality of events for the time-series data included in the two-dimensional motion data based on conditions related to the operation states of a person or a tool used by a person preset for each of the plurality of events.

[0120] Further, the event timing setting unit 3008 may combine manual setting and automatic setting. For example, when a manual setting input has been made for a target event, the event timing setting unit 3008 evaluates the difference between the timing of the manual setting input and the timing set automatically. Then, when the difference is relatively small with respect to a predetermined criterion, the event timing setting unit 3008 may adopt the timing set automatically, and in other cases, may adopt the timing of the manual setting input. Being relatively small with respect to a predetermined criterion may mean being less than or equal to the predetermined criterion, or may mean being smaller than the predetermined criterion.

[0121] For example, when there is two-dimensional motion data for each of a plurality of viewpoints, the event timing setting unit 3008 can perform time-series alignment of two-dimensional motion data with different viewpoints based on the timing of events set in the time-series data. Therefore, the three-dimensional motion data estimation unit 3005 can more accurately estimate three-dimensional motion data based on data from a plurality of different viewpoints.

[0122] Also, when reference data is used, the event timing setting unit 3008 can perform time-series alignment of the reference data and the two-dimensional motion data based on the timing of events preset in the reference data and the timing of events set in the two-dimensional motion data. Further, the event timing setting unit 3008 can equalize the number of time-series data between adjacent events between the reference data and the two-dimensional motion data. Therefore, the three-dimensional motion data estimation unit 3005 can more accurately estimate three-dimensional motion data.

[0123] [Second Example of the Operation of the Information Processing System] Next, with reference to FIG. 12, a second example of the operation of the information processing system 1 will be described.

[0124] FIG. 12 is a sequence diagram schematically showing a second example of the operation of the information processing system 1.

[0125] As shown in FIG. 12, the processing from step S202 to step S208 is the same as the processing from step S102 to step S108 in FIG. 10 except that moving image data from two different viewpoints is acquired, and thus the description thereof is omitted.

[0126] After the completion of the processing in step S208, the two-dimensional motion data acquisition unit 3002 acquires two-dimensional motion data for two viewpoints by extracting feature points representing the user's motion in each frame for the moving image data from each of the two viewpoints (step S210).

[0127] After the completion of the process in step S210, the event timing setting unit 3008 sets the timings of a plurality of events for each of the two-dimensional motion data of the two viewpoints (step S212).

[0128] After the completion of the process in step S212, the event timing setting unit 3008 compares the timings of the events set for each of the two-dimensional motion data of the two viewpoints, and aligns the time-series data of the two-dimensional motion data of the two viewpoints (step S214).

[0129] After the completion of the process in step S214, the simulation data acquisition unit 3004 acquires three-dimensional simulation data using the simulator unit 3003 (step S216).

[0130] After the completion of the process in step S216, the simulation data acquisition unit 3004 acquires two-dimensional simulation data of the same two viewpoints as the two-dimensional motion data based on the three-dimensional simulation data (step S218).

[0131] The processes from step S220 to step S228 after the completion of the process in step S218 are the same as the processes from step S116 to step S124 in FIG. 10 described above, so the description is omitted.

[0132] [Third Example of the Operation of the Information Processing System] Next, with reference to FIG. 13, a third example of the operation of the information processing system 1 will be described.

[0133] FIG. 13 is a sequence diagram schematically showing a third example of the operation of the information processing system 1.

[0134] As shown in FIG. 13, the processes from step S302 to step S310 are the same as the processes from step S102 to step S110 in FIG. 10 described above, so the description is omitted.

[0135] After the completion of the process in step S310, the event timing setting unit 3008 sets the timings of a plurality of events for the two-dimensional motion data (step S312).

[0136] After the completion of the process in step S312, the event timing setting unit 3008 compares the timings of the plurality of events set for each of the two-dimensional motion data and the reference data, and performs alignment of the time-series data (step S314).

[0137] Since the processes after step S316 after the completion of the process in step S314 are the same as the processes after step S112 in FIG. 10 described above, the description thereof is omitted.

[0138] [Fourth Example of the Operation of the Information Processing System] FIG. 14 is a sequence diagram schematically showing a fourth example of the operation of the information processing system 1.

[0139] As shown in FIG. 14, since the processes from step S402 to step S418 are the same as the processes from step S302 to step S318 in FIG. 13 described above, the description thereof is omitted.

[0140] After the completion of step S418, based on the data acquired in steps S410 and S418, the three-dimensional motion data estimation unit 3005 determines the dimensions of the rigid body links of the body model using the data assimilation process, with the dimensions of the rigid body links of the body model as variables (step S420).

[0141] After the completion of the process in step S420, the simulation data acquisition unit 3004 reflects the determined dimensions of the rigid body links in step S418 in the body model as fixed values, and acquires three-dimensional simulation data using the simulator unit 3003 (step S422).

[0142] Note that the dimensions of the rigid body links of the body model used in the numerical simulation in the process of step S416 correspond to provisional values.

[0143] After the completion of the process in step S422, the simulation data acquisition unit 3004 acquires two-dimensional simulation data based on the three-dimensional simulation data acquired in step S422 (step S424).

[0144] After the completion of the process in step S424, based on the data acquired in steps S410 and S424, the three-dimensional motion data estimation unit 3005 estimates three-dimensional motion data representing the motion of a person corresponding to the two-dimensional motion data using the data assimilation process without using the dimensions of the rigid links of the body model as variables (step S426).

[0145] After the completion of the process in step S426, the processes after step S426 are the same as the processes after step S118 in FIG. 10 described above, so the description is omitted.

[0146] [Other Embodiments] Next, other embodiments will be described.

[0147] In the above-described embodiments, modifications and changes may be appropriately made.

[0148] For example, in the above-described embodiments, the functions of the user terminal 200 and the information processing apparatus 300 may be realized by one information processing apparatus, or may be realized in a distributed manner by three or more information processing apparatuses.

[0149] Also, in the above-described embodiments and examples of their modifications and changes, two-dimensional motion data may be acquired based on information different from the moving image data of the camera 100.

[0150] Further, in the above-described embodiments and examples of their modifications and changes, three-dimensional motion data representing the user's motion may be estimated using information on the depth direction of a two-dimensional moving image. For example, based on both the three-dimensional motion data estimated using data assimilation processing and the three-dimensional motion data estimated using information on the depth direction of a two-dimensional moving image, three-dimensional motion data representing the user's motion is estimated. In this case, the final three-dimensional motion data may be obtained without giving priority to the two three-dimensional motion data, or the final three-dimensional motion data may be obtained in such a form that one of them is corrected by the other on the premise of either one of them.

[0151] [Operation] Next, the operation of the information processing method, program, and information processing apparatus according to the present embodiment will be described.

[0152] In the first aspect of the present embodiment, the information processing method is executed by an information processing apparatus and includes a first acquisition step, a second acquisition step, and an estimation step. The information processing apparatus is, for example, the above-described information processing apparatus 300. The first acquisition step is, for example, the above-described step S110, step S210, step S310, or step S410. The second acquisition step is, for example, the above-described step S112, step S216, step S316, or step S422. The estimation step is, for example, the above-described step S116, step S220, step S320, or step S426. Specifically, in the first acquisition step, observation data representing a person's motion in two dimensions is acquired. Further, in the second acquisition step, data of a numerical simulation obtained by simulating a person's motion in three dimensions is acquired. Then, in the estimation step, based on the observation data acquired in the first acquisition step and the numerical simulation data acquired in the second acquisition step, using data assimilation processing, data representing the person's motion corresponding to the observation data acquired in the first acquisition step in three dimensions is estimated.

[0153] Also, in the first aspect of the present embodiment, the program may cause the information processing apparatus to execute a first acquisition step, a second acquisition step, and an estimation step.

[0154] Also, in the first aspect of the present embodiment, the information processing apparatus may include a first acquisition unit, a second acquisition unit, and an estimation unit. The first acquisition unit is, for example, the two-dimensional motion data acquisition unit 3002 described above. The second acquisition unit is, for example, the simulation data acquisition unit 3004 described above. The estimation unit is, for example, the three-dimensional motion data estimation unit 3005 described above. Specifically, the first acquisition unit acquires observation data representing a person's motion in two dimensions. Also, the second acquisition unit acquires data of a numerical simulation that simulates a person's motion in three dimensions. Then, based on the observation data acquired by the first acquisition unit and the data of the numerical simulation acquired by the second acquisition unit, the estimation unit uses data assimilation processing to estimate data representing the person's motion corresponding to the observation data acquired by the first acquisition unit in three dimensions.

[0155] Thereby, the information processing apparatus can acquire data representing a person's motion corresponding to the observation data in three dimensions from the observation data representing the person's motion in two dimensions and the data of the numerical simulation that simulates the person's motion in three dimensions. Therefore, for example, when performing three-dimensional measurement of a person's motion, it is not necessary to use a plurality of measuring devices such as cameras, or to attach markers to the target person or the person's tools. Also, for example, it is not necessary to require a large amount of teacher data as in the case of estimating data representing a person's motion in three dimensions from an image using supervised learning. Thus, the information processing apparatus can relatively easily acquire data representing a person's motion in three dimensions.

[0156] Also, in the second aspect of the present embodiment, on the premise of the above-described first aspect, in the estimation step, with respect to the data of the numerical simulation acquired in the second acquisition step, by the data assimilation process of reflecting the observation data acquired in the first acquisition step, data representing a person's motion corresponding to the observation data acquired in the first acquisition step may be estimated in three dimensions.

[0157] Thereby, the information processing apparatus can acquire data representing a person's motion in three dimensions by reflecting the observation data on the data of the numerical simulation that simulates a person's motion in three dimensions.

[0158] Also, in the third aspect of the present embodiment, on the premise of the above-described second aspect, in the data assimilation process, a Bayesian filter based on Bayes' theorem may be applied.

[0159] Thereby, the information processing apparatus can reflect the observation data on the data of the numerical simulation that simulates a person's motion in three dimensions by filtering with a Bayesian filter.

[0160] Also, in the fourth aspect of the present embodiment, on the premise of the above-described third aspect, the Bayesian filter may be an extended Kalman filter.

[0161] Thereby, the information processing apparatus can relatively small suppress the amount of calculation in the data assimilation process. This is because the amount of calculation of the extended Kalman filter is generally relatively small compared to other types of Bayesian filters.

[0162] Further, in the fifth aspect of the present embodiment, on the premise of any one of the above-described first to third aspects, in the first acquisition step, observation data representing a specific motion of a person in two dimensions may be acquired. The specific motion is, for example, the above-described golf swing motion, baseball batting motion, or tennis swing motion. And in the second acquisition step, based on the reference data representing the specific motion of the person, data of the numerical simulation that simulates the motion of the person in three dimensions may be acquired.

[0163] Thereby, the information processing apparatus can estimate data representing a specific motion of a person in three dimensions based on the reference data corresponding to the specific motion of the person. Therefore, the information processing apparatus can more accurately estimate data representing a specific motion of a person in three dimensions.

[0164] Further, in the sixth aspect of the present embodiment, on the premise of any one of the above-described first to fourth aspects, in the first acquisition step, observation data representing the motion of a person in two dimensions as seen from a plurality of viewpoints may be acquired.

[0165] Thereby, the information processing apparatus can estimate data representing the motion of a person in three dimensions using the observation data representing the motion of the person as seen from a plurality of viewpoints. Therefore, the information processing apparatus can more accurately estimate data representing the motion of a person in three dimensions.

[0166] Further, in the seventh aspect of the present embodiment, on the premise of any one of the above-described first to fifth aspects, the observation data acquired in the first acquisition step and the data of the numerical simulation acquired in the second acquisition step may include data representing a body part of a person and data representing a tool held by the person. And the data representing the motion of the person in three dimensions corresponding to the observation data acquired in the first acquisition step, which is estimated in the estimation step, may include data representing the state of the body part of the person in three dimensions and data representing the tool held by the person in three dimensions.

[0167] As a result, the information processing apparatus can estimate data representing a person's motion in three dimensions, taking into account not only the motion of the person's body parts but also the motion of the tools held by the person. Therefore, the information processing apparatus can estimate data representing a person's motion in three dimensions with higher accuracy.

[0168] Also, in the eighth aspect of the present embodiment, on the premise of any one of the first to sixth aspects described above, in the first acquisition step, observation data representing the positions of the person's body feature points in two dimensions in time series may be acquired based on a moving image representing the person's motion.

[0169] As a result, the information processing apparatus can estimate data representing the person's motion in three dimensions from a moving image, which is two-dimensional information representing the person's motion.

[0170] Also, in the ninth aspect of the present embodiment, on the premise of any one of the first to seventh aspects described above, the first acquisition step may include a third acquisition step of acquiring the timings of a plurality of events related to the specific motion for the observation data representing the person's specific motion in two dimensions acquired in the first acquisition step. The third acquisition step is, for example, the above-described step S212, step S312, or step S412.

[0171] As a result, the information processing apparatus can perform time-series alignment between the observation data representing the person's specific motion in two dimensions and the data of a numerical simulation simulating the person's specific motion in three dimensions. Therefore, the information processing apparatus can estimate data representing the person's motion in three dimensions with higher accuracy.

[0172] Also, in the tenth aspect of the present embodiment, on the premise of the eighth aspect described above, in the third acquisition step, at least a part of the timings of the plurality of events corresponding to the observation data may be automatically acquired based on the observation data acquired in the first acquisition step.

[0173] As a result, the information processing apparatus can automate the acquisition of the timings of a plurality of events.

[0174] Further, in the eleventh aspect of the present embodiment, on the premise of the above-described eighth or ninth aspect, in the second acquisition step, based on reference data representing the specific motion of a person, data of the numerical simulation obtained by simulating the motion of the person in three dimensions may be acquired. And in the reference data, the timings of the plurality of events may be defined in advance.

[0175] As a result, the information processing apparatus can perform time-series alignment between observation data representing a specific motion of a person in two dimensions and data of a numerical simulation obtained by simulating the specific motion of the person in three dimensions.

[0176] Further, in the twelfth aspect of the present embodiment, on the premise of any one of the above-described first to tenth aspects, in the second acquisition step, taking the dimensions of a predetermined part of the body model used in the numerical simulation as provisional values, first data of the numerical simulation may be acquired. The dimensions of a predetermined part of the body model are, for example, the dimensions of the rigid links connecting the joint parts of the above-described body model MD1. Also, in the estimation step, based on the observation data acquired in the first acquisition step and the first data of the numerical simulation acquired in the second acquisition step, using the data assimilation process, data representing the dimensions of a predetermined part of the body model used in the numerical simulation corresponding to the observation data acquired in the first acquisition step may be determined. Further, in the second acquisition step, based on the body model in which the dimensions of the predetermined part determined in the estimation step are reflected, second data of the numerical simulation may be acquired. And in the estimation step, based on the observation data acquired in the first acquisition step and the second data of the numerical simulation acquired in the second acquisition step, again using the data assimilation process, data representing the motion of the person corresponding to the observation data acquired in the first acquisition step in three dimensions may be estimated.

[0177] As a result, the information processing apparatus can, by the data assimilation process, estimate, as variables (parameters), the dimensions of predetermined parts of the body model used in numerical simulation, as well as the data representing human movements, and make a determination based on the estimation results. Therefore, the information processing apparatus can fix the dimensions of the predetermined parts of the determined body model and perform the data assimilation process again, thereby more accurately estimating the data representing human movements in three dimensions.

[0178] As described in detail above for the embodiments, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

Description of Reference Numerals

[0179] 1 Information processing system 100 Camera 200 User terminal 300 Information processing apparatus 2001 Application screen display processing unit 2002 Moving image data acquisition unit 2003 Moving image data transmission unit 2004 Analysis result data acquisition unit 3001 Moving image data acquisition unit 3002 Two-dimensional motion data acquisition unit 3003 Simulator unit 3004 Simulation data acquisition unit 3005 Three-dimensional motion data estimation unit 3006 Motion analysis unit 3007 Analysis result transmission unit 3008 Event timing setting unit

Claims

1. A first acquisition step in which an information processing apparatus acquires observation data representing a person's motion in two dimensions; A second acquisition step in which an information processing apparatus acquires data of a numerical simulation that simulates a person's motion in three dimensions; An estimation step in which the information processing apparatus estimates data representing a person's motion corresponding to the observation data acquired in the first acquisition step in three dimensions by using data assimilation processing based on the observation data acquired in the first acquisition step and the data of the numerical simulation acquired in the second acquisition step. An information processing method.

2. In the estimation step, data representing a person's motion corresponding to the observation data acquired in the first acquisition step is estimated in three dimensions by the data assimilation processing that reflects the observation data acquired in the first acquisition step on the data of the numerical simulation acquired in the second acquisition step. The information processing method according to Claim 1.

3. In the data assimilation processing, a Bayesian filter based on Bayes' theorem is applied. The information processing method according to Claim 2.

4. The Bayesian filter is an extended Kalman filter. The information processing method according to Claim 3.

5. In the first acquisition step, observation data representing a specific motion of a person in two dimensions is acquired. In the second acquisition step, data of the numerical simulation that simulates a person's motion in three dimensions is acquired based on reference data representing the specific motion of the person. The information processing method according to any one of Claims 1 to 4.

6. In the first acquisition step, observation data representing a person's motion when viewed from a plurality of viewpoints in two dimensions is acquired. The information processing method according to any one of Claims 1 to 4.

7. The observation data acquired in the first acquisition step and the data of the numerical simulation acquired in the second acquisition step include data representing a person's body part and data representing a tool held by the person. The data representing a person's motion corresponding to the observation data acquired in the first acquisition step, which is estimated in the estimation step, includes data representing the state of the person's body part in three dimensions and data representing a tool held by the person in three dimensions. The information processing method according to any one of Claims 1 to 4.

8. In the first acquisition step, based on a moving image representing a person's movement, observation data representing the positions of the person's physical feature points in two dimensions in time series is acquired. The information processing method according to any one of claims 1 to 4.

9. Including a third acquisition step of acquiring the timings of a plurality of events related to the specific movement for the observation data representing the specific movement of a person in two dimensions acquired in the first acquisition step. The information processing method according to any one of claims 1 to 4.

10. In the third acquisition step, based on the observation data acquired in the first acquisition step, at least a part of the timings of the plurality of events corresponding to the observation data is automatically acquired. The information processing method according to claim 9.

11. In the second acquisition step, based on reference data representing the specific movement of a person, data of the numerical simulation that simulates the person's movement in three dimensions is acquired. In the reference data, the timings of the plurality of events are defined in advance. The information processing method according to claim 9.

12. In the second acquisition step, with the dimensions of a predetermined part of the body model used in the numerical simulation as provisional values, first data of the numerical simulation is acquired. In the estimation step, based on the observation data acquired in the first acquisition step and the first data of the numerical simulation acquired in the second acquisition step, using the data assimilation process, the dimensions of the predetermined part corresponding to the observation data acquired in the first acquisition step are determined. In the second acquisition step, the dimensions of the predetermined part determined in the estimation step are Based on the body model in which the above is reflected, second data of the numerical simulation is acquired. In the estimation step, based on the observation data acquired in the first acquisition step and the second data of the numerical simulation acquired in the second acquisition step, again using the data assimilation process, data representing the person's movement in three dimensions corresponding to the observation data acquired in the first acquisition step is estimated. The information processing method according to any one of claims 1 to 4.

13. In an information processing apparatus, A first acquisition step of acquiring observation data representing a person's movement in two dimensions, and A second acquisition step of acquiring data of a numerical simulation that simulates a person's movement in three dimensions. An estimation step of estimating data representing a person's motion in three dimensions corresponding to the observation data acquired in the first acquisition step by using data assimilation processing based on the observation data acquired in the first acquisition step and the data of the numerical simulation acquired in the second acquisition step, and causing the estimation step to be executed. Program. **Claim 14** A first acquisition unit that acquires observation data representing a person's motion in two dimensions; A second acquisition unit that acquires data of a numerical simulation that simulates a person's motion in three dimensions; An estimation unit that estimates data representing a person's motion in three dimensions corresponding to the observation data acquired by the first acquisition unit by using data assimilation processing based on the observation data acquired by the first acquisition unit and the data of the numerical simulation acquired by the second acquisition unit. An information processing apparatus.

Images