Information processing apparatus, information processing method, and program

Abstract

An information processing system that includes circuitry configured to acquire sensor data indicating motion data of at least one actor and motion data of at least one object, synchronize the motion data of the at least one actor and the motion data of the at least one object, detect a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object, and estimate a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

Description

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2023-213727 filed December 19, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

[0003] In recent years, a technology for acquiring motion data relating to a user's operation using a sensor have been developed. Further, as disclosed in, for example, Patent Literature 1, there is a technology for detecting an interaction between a user and an object different from the user.

[0004] Japanese Patent Application Laid-open No. 2016-208516Summary

[0005] However, the technology disclosed in Patent Literature 1 only detects the above interaction and does not reproduce the interaction. Further, in the case of attempting to reproduce an interaction with a user and an object, highly-accurate sensing is generally required, which increases the cost of equipment and calculation.

[0006] According to the present disclosure, there is provided an information processing system that includes circuitry configured to acquire sensor data indicating motion data of at least one actor and motion data of at least one object, synchronize the motion data of the at least one actor and the motion data of the at least one object, detect a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object, and estimate a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

[0007] Furthermore, according to the present disclosure, an information processing method includes acquiring sensor data indicating motion data of at least one actor and motion data of at least one object, synchronizing the motion data of the at least one actor and the motion data of the at least one object, detecting a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object, and estimating a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

[0008] In addition, according to the present disclosure, a non-transitory computer-readable medium is provided having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method including acquiring sensor data indicating motion data of at least one actor and motion data of at least one object, synchronizing the motion data of the at least one actor and the motion data of the at least one object, detecting a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object, and estimating a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

[0009] Fig. 1 is a diagram describing an overview of an information processing method according to an embodiment of the present disclosure.Fig. 2 is a block diagram showing an example of a functional configuration of an information processing system 1 according to an embodiment.Fig. 3 is a block diagram showing an example of a functional configuration of a processing unit 320 according to an embodiment.Fig. 4A is a diagram showing an example of data input / output in the processing unit 320 according to an embodiment.Fig. 4B is a diagram showing an example of data input / output in the processing unit 320 according to an embodiment.Fig. 5 is a diagram describing time synchronization in an embodiment.Fig. 6 is a diagram describing detection of a time of contact in an embodiment.Fig. 7 is a diagram describing an example of bone information in an embodiment.Fig. 8 is a diagram showing a detailed example of joints specified in the bone information in an embodiment.Fig. 9 is a flowchart showing an example of the flow of contact position estimation using a rule-based method in an embodiment.Fig. 10 is a diagram showing the flow of data for contact position estimation using a machine learning method in an embodiment.Fig. 11 is a diagram describing estimation of the trajectory of a virtual object 74 in an embodiment.Fig. 12 is a flowchart showing an example of the flow of reproduction of lifting in an embodiment.Fig. 13 is a diagram describing a case where the information processing method according to an embodiment is applied to reproduction of a volleyball match.Fig. 14 is a diagram describing a case where the information processing method according to an embodiment is applied to reproduction of a table tennis match.Fig. 15 is a diagram showing an example of presentation of an operation analysis result in an embodiment.Fig. 16 is a diagram describing a case where the information processing method according to an embodiment is applied to reproduction of juggling.Fig. 17 is a diagram describing rendering of an interaction, which is difficult in reality, in an embodiment.Fig. 18 is a block diagram showing a hardware configuration example of an information processing apparatus 90 according to an embodiment.

[0010] A suitable embodiment of the present disclosure will be described below in detail with reference to the accompanying drawings. Note that in the present specification and drawings, components having substantially the same functional configuration will be denoted by the same reference symbols, and duplicate description thereof will be omitted.

[0011] Further, in the present specification and drawings, in the case where a plurality of similar configurations is to be distinguished and described, an alphabetic character or the like is added to the end of the reference symbol in some cases. Meanwhile, in the case where there is no need to distinguish the plurality of similar configurations, the above alphabetic character or the like is omitted and description common to the plurality of similar configurations is made in some cases.

[0012] Note that description will be made in the following order. 1. Embodiment 1.1. Overview 1.2. functional configuration example 1.3. Detailed function 1.4. Application example 2. Hardware configuration example 3. Conclusion

[0013] <1. Embodiment> <<1.1. Overview>> As described above, in the case of trying to reproduce an interaction between an actor such as a user and an object different from the actor, it is necessary to measure motions of the actor and the object with high accuracy and perform rendering based on the measurement result.

[0014] Examples of the method of measuring motions of the actor and the object with high accuracy include optical motion capture that performs measurement with markers attached to the actor and the object and video motion capture that measures the bones of the actor and the behavior of the object from a plurality of moving images.

[0015] However, in the case of performing measurement using the above method, high costs are necessary for measurement equipment.

[0016] Further, in the case of performing measurement using the above method, sufficient calculation resources are necessary to process motion data measured with high accuracy.

[0017] Further, in the case of performing measurement using the above method, since the environment in which measurement equipment can be installed is limited, the environment in which measurement can be performed is also limited.

[0018] The technical idea according to an embodiment of the present disclosure has been conceived in light of the above point, and it is desired to realize an interaction between an actor and an object with a simpler configuration.

[0019] Fig. 1 is a diagram describing an overview of an information processing method according to an embodiment of the present disclosure.

[0020] A measurement environment in a real space 50 is illustrated on the left side of Fig. 1.

[0021] In the real space 50, a first sensor group 10N including N first sensors 10 is attached to an actor 52 such as a user.

[0022] Note that although a case where the first sensor group 10N includes the following first sensors 10a to 10f is illustrated in Fig. 1, the number of first sensors 10 included in the first sensor group 10N and the attachment positions thereof are not limited to such an example.

[0023] First sensor 10a: attached to the head of the actor 52 First sensor 10b: attached to the hip of the actor 52 First sensor 10c: attached to the right wrist of the actor 52 First sensor 10d: attached to the left wrist of the actor 52 First sensor 10e: attached to right ankle of the actor 52 First sensor 10f: attached to left ankle of the actor 52

[0024] The first sensor group 10N acquires motion data of the actor 52. Each of the first sensors 10 included in the first sensor group 10N may be an inertial measurement unit (IMU).

[0025] Further, in the real space 50, at least one second sensor 20 is attached to an object 54 different from the actor 52.

[0026] For example, in the case where the object 54 is a ball, the second sensor 20 may be embedded inside the ball.

[0027] The second sensor 20 acquires motion data of the object 54. The second sensor 20 may be an inertial measurement unit.

[0028] In the information processing method according to an embodiment of the present disclosure, as shown on the right side of Fig. 1, an avatar 72 and a virtual object 74 in a virtual space 70 are drawn on the basis of the motion data of the actor 52 and the motion data of the object 54 acquired as described above.

[0029] The avatar 72 is computer graphics (CG) corresponding to the actor 52. Further, the virtual object 74 is CG corresponding to the object 54.

[0030] A processor that executes the information processing method according to an embodiment of the present disclosure may draw the avatar 72 and the virtual object 74 such that an interaction between the actor 52 and the object 54 in the real space 50 is reproduced.

[0031] In order to achieve this, the information processing method according to an embodiment of the present disclosure includes synchronizing times of the motion data of the object 54 in the real space 50 and the motion data of the actor 52 in the real space 50.

[0032] Further, the information processing method according to an embodiment of the present disclosure further includes detecting a time of contact between the object 54 and the actor 52 on the basis of the synchronized motion data of the object 54 and the synchronized motion data of the actor 52.

[0033] Further, the information processing method according to an embodiment of the present disclosure further includes estimating a contact position between the object 54 and the actor 52 on the basis of the synchronized motion data of the object 54, the synchronized motion data of the actor 52, and the time of contact.

[0034] Further, the information processing method according to an embodiment of the present disclosure may further include rendering an interaction between the avatar 72 and the virtual object 74 in the virtual space 70 on the basis of the time of contact and the contact position.

[0035] In accordance with the above configuration, it is possible to realize an interaction between the actor 52 and the object 54 with a simpler configuration.

[0036] A functional configuration for executing the information processing method according to an embodiment of the present disclosure will be described below as an example.

[0037] <<1.2. Functional configuration example>> Fig. 2 is a block diagram showing a functional configuration example of an information processing system 1 according to an embodiment of the present disclosure.

[0038] As shown in Fig. 2, the information processing system 1 according to this embodiment may include the first sensor group 10N, the at least one second sensor 20, and an information processing apparatus 30.

[0039] (First sensor group 10N) As described above, the first sensor group 10N according to this embodiment includes N first sensors 10.

[0040] Each of the N first sensors 10 is attached to a predetermined part of the body of the actor 52.

[0041] Note that although a case where the actor 52 is a person is assumed as a main example in this embodiment, the actor 52 is not limited to such an example.

[0042] The actor 52 may be an animal other than humans or may be an automated moving object.

[0043] (First sensor 10) The first sensor 10 according to this embodiment acquires motion data of the actor 52. The first sensor 10 according to this embodiment may be an inertial measurement unit.

[0044] As shown in Fig. 2, the first sensor 10 according to this embodiment includes a wireless communication unit 110 and a sensor unit 120.

[0045] (Wireless communication unit 110) The wireless communication unit 110 performs wireless communication conforming to a predetermined communication standard with the information processing apparatus 30.

[0046] For example, the wireless communication unit 110 transmits the motion data acquired by the sensor unit 120 to the information processing apparatus 30.

[0047] Examples of the above predetermined communication standard include Bluetooth (registered trademark) Low Energy (BLE). However, the above predetermined communication standard is not limited to such an example.

[0048] (Sensor unit 120) The sensor unit 120 acquires motion data of the actor 52.

[0049] The motion data according to this embodiment may include, for example, 3-axis acceleration and 3-axis angular velocity.

[0050] The sensor unit 120 includes various sensors for acquiring motion data, such as an acceleration sensor and a gyro sensor.

[0051] (Second sensor 20) The second sensor 20 according to this embodiment is attached to a predetermined part of the object 54.

[0052] Note that although a case where the object 54 is a ball is assumed as a main example in this embodiment, the object 54 is not limited to such an example. The object 54 may be an arbitrary object.

[0053] As shown in Fig. 2, the second sensor 20 according to this embodiment includes a wireless communication unit 210 and a sensor unit 220.

[0054] (Wireless communication unit 210) The wireless communication unit 210 performs wireless communication conforming to a predetermined communication standard with the information processing apparatus 30.

[0055] For example, the wireless communication unit 210 transmits the motion data acquired by the sensor unit 220 to the information processing apparatus 30.

[0056] (Sensor unit 220) The sensor unit 220 acquires motion data of the object 54,

[0057] The sensor unit 220 includes various sensors for acquiring motion data, such as an acceleration sensor and a gyro sensor.

[0058] (Information processing apparatus 30) The information processing apparatus 30 is a computer for executing the information processing method according to this embodiment.

[0059] As shown in Fig. 2, the information processing apparatus 30 according to this embodiment may include a wireless communication unit 310, a processing unit 320, and a display unit 330.

[0060] (Wireless communication unit 310) The wireless communication unit 310 performs wireless communication conforming to a predetermined communication standard with each of the first sensor 10 and the second sensor 20.

[0061] For example, the wireless communication unit 310 receives motion data from each of the first sensor 10 and the second sensor 20.

[0062] (Processing unit 320) The processing unit 320 according to this embodiment executes various types of processing on the basis of the motion data of the actor 52 and the motion data of the object 54 received by the wireless communication unit 310.

[0063] The function of the processing unit 320 according to this embodiment is realized by various processors.

[0064] Fig. 3 is a block diagram showing a functional configuration example of the processing unit 320 according to this embodiment.

[0065] As shown in Fig. 3, the processing unit 320 according to this embodiment may include a calibration unit 321, a time synchronization unit 322, a contact time detection unit 323, a contact position estimation unit 324, and a rendering unit 325.

[0066] (Calibration unit 321) The calibration unit 321 according to this embodiment executes calibration for the first sensor group 10N.

[0067] In the calibration in this embodiment, in order to achieve more accurate motion capture, the orientation of each of the first sensors 10, the height of the actor 52, and the like are acquired, and various adjustments are made on the basis of the various acquired information.

[0068] The execution of the above calibration allows bone information of the actor 52 described below to be acquired.

[0069] (Time synchronization unit 322) The time synchronization unit 322 according to this embodiment synchronizes times of motion data of the object 54 in the real space 50 and motion data of the actor 52 in the real space 50.

[0070] Details of the function of the time synchronization unit 322 according to this embodiment will be described below.

[0071] (Contact time detection unit 323) The contact time detection unit 323 according to this embodiment detects the time at which the object 54 and the actor 52 came into contact with each other (time of contact) on the basis of the synchronized motion data of the object 54 and the synchronized motion data of the actor 52.

[0072] Details of the function of the contact time detection unit 323 according to this embodiment will be described below.

[0073] (Contact position estimation unit 324) The contact position estimation unit 324 according to this embodiment estimates the position at which the object 54 and the actor 52 came into contact with each other (contact position) on the basis of the synchronized motion data of the object 54, the synchronized motion data of the actor 52, and the time of contact.

[0074] Details of the function of the contact position estimation unit 324 according to this embodiment will be described below.

[0075] (rendering unit 325) The rendering unit 325 according to this embodiment renders an interaction between the avatar 72 corresponding to the actor 52 the virtual space 70 and the virtual object 74 corresponding to the object 54 in the virtual space 70 on the basis of the time of contact and the contact position.

[0076] Details of the function of the rendering unit 325 according to this embodiment will be described below.

[0077] (Display unit 330) The display unit 330 according to this embodiment displays various types of information in accordance with control of the processing unit 320.

[0078] For example, the display unit 330 according to this embodiment displays the result of the rendering by the rendering unit 325.

[0079] The display unit 330 includes various displays.

[0080] A functional configuration example of the information processing system 1 according to this embodiment has been described above. Note that the above functional configuration described with reference to Fig. 2 and Fig. 3 is merely an example, and the functional configuration of the information processing system 1 according to this embodiment is not limited to such an example.

[0081] For example, the above-mentioned calibration, acquisition of bone information based on the calibration, and the like may be performed by a device (such as a smartphone) separate from the information processing apparatus 30.

[0082] Further, for example, the time synchronization, detection of a time of contact, contact position estimation, rendering, and displaying may be realized by cooperation between a plurality of devices.

[0083] The functional configuration of the information processing system 1 according to this embodiment can be flexibly modified in accordance with the specifications, operations, and the like. Examples of modifications to the information processing system according to an embodiment of the present disclosure include a system that only includes the information processing apparatus, a system that includes both the information processing apparatus and the first sensor(s), a system that includes both the information processing apparatus and the second sensor(s), or a system that includes all elements (the information processing apparatus, one or more first sensors, and one or more second sensors), but is not limited thereto.

[0084] <<1.3. Detailed function>> Next, the function of the processing unit 320 according to this embodiment will be described in detail. In the following, as an interaction between the actor 52 and the object 54, lifting in soccer or the like will be described as a main example.

[0085] Further, in the following, description will be made with reference to Fig. 4A and Fig. 4B. Fig. 4A and Fig. 4B are each a diagram showing an example of data input / output in the processing unit 320 according to this embodiment.

[0086] First, time synchronization by the time synchronization unit 322 according to this embodiment will be described.

[0087] As shown in Fig. 4A, the motion data of the actor 52 and the motion data of the object 54 are input to the time synchronization unit 322 according to this embodiment.

[0088] The motion data of the actor 52 to be input to the time synchronization unit 322 may include the bone information acquired on the basis of the acceleration and angular velocity sensed by the first sensor group 10N.

[0089] Further, the motion data of the object 54 to be input to the time synchronization unit 322 includes at least acceleration.

[0090] The time synchronization unit 322 performs time synchronization on the basis of the input data, and outputs the motion data of the actor 52 after time synchronization and the motion data of the object 54 after time synchronization.

[0091] Fig. 5 is a diagram describing time synchronization in this embodiment.

[0092] Fig. 5 shows images of the motion data of the object 54 and the motion data of the actor 52 acquired in time series.

[0093] In Fig. 5, the motion data of the object 54 in the 0-th frame before time synchronization is indicated by a circle of dots, and the motion data of the actor 52 in the 0-th frame before time synchronization is indicated by a shaded circle.

[0094] Further, in Fig. 5, the motion data of the object 54 in the 0-th frame after time synchronization and the motion data of the actor 52 in the 0-th frame after time synchronization are each indicated by a white circle.

[0095] In the time synchronization in this embodiment, for example, before lifting measurement, an instruction is given to the actor 52 to jump with the object 54, via the display unit 330 or the like.

[0096] In this case, the time synchronization unit 322 synchronizes, after the instruction is given, the time at which an impact exceeding a threshold value was detected on the object 54 and the time at which it was detected that both feet of the actor 52 were off the ground and then came into contact with the ground again.

[0097] The time synchronization unit 322 is capable of detecting the time at which an impact exceeding a threshold value occurred on the object 54 (indicated by a white arrow) on the basis of the acceleration of the object 54, or the like.

[0098] For example, the time synchronization unit 322 may use, in the case where the time at which an impact exceeding a threshold value was detected on the object 54 is the 171-st frame before time synchronization, this frame as the 0-th frame after time synchronization.

[0099] Further, the time synchronization unit 322 is capable of detecting the time at which both feet of the actor 52 were off the ground (indicated by a symbol obtained by combining a double circle and a cross) and then came into contact with the ground again on the basis of the bone information of the actor 52, or the like.

[0100] For example, the time synchronization unit 322 may use, in the case where the time at which it was detected that both feet of the actor 52 were off the ground and then came into contact with the ground again is the 130-th frame before time synchronization, this frame as the 0-th frame after time synchronization.

[0101] In accordance with the processing as described above, it is possible to synchronize acquisition times of the motion data of the actor 52 and the motion data of the object 54. However, the time synchronization is not limited to the above-mentioned processing, and an arbitrary method may be adopted.

[0102] Next, detection of a time of contact by the contact time detection unit 323 according to this embodiment twill be described.

[0103] As shown in Fig. 4A, the motion data of the actor 52 after time synchronization and the motion data of the object 54 after time synchronization are input to the contact time detection unit 323 according to this embodiment.

[0104] The contact time detection unit 323 according to this embodiment may detect a time of contact on the basis of the input data, and output the motion data of the actor 52 after time synchronization, the motion data of the object 54 after time synchronization, and the detected time of contact.

[0105] Fig. 6 is a diagram describing detection of a time of contact in this embodiment.

[0106] Fig. 6 shows images of the motion data of the object 54 after time synchronization acquired in time series.

[0107] In an interaction involving physical contact, such as lifting, assumption is made that an impact corresponding to the interaction occurs on the object 54 at the time at which the interaction took place.

[0108] For this reason, the contact time detection unit 323 according to this embodiment may detect, as a time of contact, the time at which an impact exceeding a threshold value occurred on the object 54 after time synchronization (indicated by a shaded arrow).

[0109] In Fig. 6, the motion data of the object 54 in the 0-th frame after time synchronization is indicated by a white circle. Further, in Fig. 6, the motion data of the object 54 in the N-th frame detected as a time of contact is indicated by a double circle.

[0110] Further, the contact time detection unit 323 according to this embodiment may estimate the period between the detected times of contact as a hang time of the object 54 in lifting.

[0111] The estimated hang time is used for rendering the virtual object 74 by the rendering unit 325. Note that the hang time may be estimated by the rendering unit 325.

[0112] Next, contact position estimation by the contact position estimation unit 324 according to this embodiment will be described.

[0113] As shown in Fig. 4A, the motion data of the actor 52 after time synchronization, the motion data of the object 54 after time synchronization, and the detected time of contact are input to the contact position estimation unit 324 according to this embodiment.

[0114] The contact position estimation unit 324 according to this embodiment may perform contact position estimation on the basis of the input data, and output the motion data of the actor 52 after time synchronization, the motion data of the object 54 after time synchronization, the time of contact, and the result of contact position estimation.

[0115] For example, the contact position estimation unit 324 according to this embodiment may estimate, as a contact position, a body part (referred to also as a contact part) of the actor 52 that has come into contact with the object 54.

[0116] In this case, the contact position estimation unit 324 according to this embodiment outputs the name of the contact part (or information capable of identifying the contact part) and the coordinates of the contact part to the rendering unit 325.

[0117] First, the contact position estimation unit 324 according to this embodiment acquires three-dimensional coordinates (referred to simply also as coordinates) of each joint from the bone information included in the motion data of the actor 52 on the basis of the time of contact.

[0118] Fig. 7 is a diagram describing an example of bone information in this embodiment.

[0119] In the case of an example shown in Fig. 7, the bone information includes information relating to a plurality of joints J0 to J26 and a plurality of bones connecting between the joints.

[0120] Fig. 8 is a diagram showing a detailed example of joints specified in the bone information in this embodiment.

[0121] The Index in Fig. 8 corresponds to the numbered portion of the reference symbol shown in Fig. 7.

[0122] As shown in Fig. 8, the bone information in this embodiment may include the Index for identifying the specified joint, the name, and the like.

[0123] Further, the bone information in this embodiment includes three-dimensional coordinates of the joint estimated from the acceleration and angular velocity acquired by the first sensor group 10N.

[0124] The contact position estimation unit 324 according to this embodiment acquires three-dimensional coordinates of a plurality of joints at the time of contact and estimates the contact position on the basis of the positional relationship between the joints.

[0125] The contact position estimation unit 324 according to this embodiment may estimate, for example, a contact position using a rule-based method.

[0126] Fig. 9 is a flowchart showing an example of the flow of contact position estimation using a rule-based method in this embodiment.

[0127] First, the contact position estimation unit 324 acquires coordinates of a plurality of joints at the time of contact (S101).

[0128] In an example shown in Fig. 9, the contact position estimation unit 324 acquires coordinates of 10 joints indicated by dots in Fig. 8.

[0129] Next, the contact position estimation unit 324 determines, on the basis of the coordinates of 10 joints acquired in Step S101, whether or not one foot of the actor 52 is off the ground (S102).

[0130] In the case where it is determined that one foot is not off the ground (S102: NO), the contact position estimation unit 324 then determines, on the basis of the coordinates of 10 joints acquired in Step S101, whether or not the face of the actor is facing upwards (S103).

[0131] In the case where it is determined that the face is not facing upwards (S103: NO), the contact position estimation unit 324 determines that the actor 52 is in contact with the object 54 with his / her head (S104) and ends the contact position estimation.

[0132] Meanwhile, in the case where it is determined that the face is facing upwards (S103: YES), the contact position estimation unit 324 determines that the time of contact has been false detection (S105) and ends the contact position estimation.

[0133] Further, in the case where it is determined that one foot is off the ground (S102: YES), the contact position estimation unit 324 identifies, on the basis of the coordinates of 10 joints acquired in Step S101, the foot that is off the ground (S106).

[0134] Next, the contact position estimation unit 324 determines, on the basis of the coordinates of 10 joints acquired in Step S101, whether or not the area formed by the toe, knee, and hip joint on the side of the foot that is off the ground is a threshold value or more (S107). More specifically, the area may be the area of a triangle with the toe, knee, and hip joint on the side of the foot that is off the ground as their apexes.

[0135] In the case where it is determined that the area formed by the toe, knee, and hip joint on the side of the foot that is off the ground is less than the threshold value (S107: NO), the contact position estimation unit 324 determines that the actor 52 is in contact with the object 54 with the instep on the side of the foot that is off the ground (S108) and ends the contact position estimation.

[0136] Meanwhile, in the case where it is determined that the area formed by the toe, knee, and hip joint on the side of the foot that is off the ground is the threshold value or more (S107: YES), the contact position estimation unit 324 then determines whether or not the distance between the ankle and hip on the side of the foot that is off the ground is a threshold value or more (S109).

[0137] In the case where it is determined that the distance between the ankle and hip on the side of the foot that is off the ground is less than the threshold value (S109: NO), the contact position estimation unit 324 determines that the actor 52 is in contact with the object 54 with the inside or outside on the side of the foot that is off the ground (S110) and ends the contact position estimation.

[0138] Meanwhile, in the case where it is determined that the distance between the ankle and hip on the side of the foot that is off the ground is the threshold value or more (S109: YES), the contact position estimation unit 324 determines that the actor 52 is in contact with the object 54 with the upper leg on the side of the foot that is off the ground (S111) and ends the contact position estimation.

[0139] As described above, the contact position estimation unit 324 according to this embodiment may estimate a contact position using a rule-based method.

[0140] Meanwhile, the contact position estimation unit 324 according to this embodiment may estimate a contact position using a machine learning method.

[0141] Fig. 10 is a diagram showing the flow of data for contact position estimation using a machine learning method in this embodiment.

[0142] In the case of performing contact position estimation using a machine learning method, as shown in Fig. 10, the contact position estimation unit 324 includes a contact position classifier 326.

[0143] The contact position classifier 326 according to this embodiment is generated in advance by, for example, supervised learning based on the pair of learning data including coordinates of all 27 joints at the time of contact and information (e.g., the name of the contact part) relating to the contact part of a correct answer (Ground Truth) given by the developer or the like to the learning data.

[0144] The contact position classifier 326 generated by the above learning is capable of outputting information relating to the contact part on the basis of the input coordinates of all 27 joints at the time of contact.

[0145] Further, the contact position estimation unit 324 according to this embodiment is capable of acquiring, on the basis of information relating to a contact part output by the contact position classifier 326, coordinates of the contact part from the bone information.

[0146] In accordance with the contact position estimation using a machine learning method in this embodiment, for example, it is possible to classify the head / upper leg / instep / inside or outside, similarly to the contact position estimation using a rule-based method.

[0147] However, the above classification is merely an example for both the contact position estimation using a rule-based method and the contact position estimation using a machine learning method, and the number of classifications and settings for the contact part can be flexibly modified.

[0148] Next, rendering of the avatar 72, the virtual object 74, and the like by the rendering unit 325 according to this embodiment will be described.

[0149] As shown in Fig. 4B, the motion data of the actor 52 after time synchronization, the motion data of the object 54 after time synchronization, the time of contact, the contact part name, and the contact part coordinates, which are input from the contact position estimation unit 324, are input to the rendering unit 325 according to this embodiment.

[0150] Further, avatar data, virtual object data, and virtual space data are input to the rendering unit 325 according to this embodiment from a storage unit (not shown).

[0151] The avatar data includes a CG material or the like necessary for rendering the avatar 72. The virtual object data includes a CG material or the like necessary for rendering the virtual object 74. The virtual space data includes a CG material or the like necessary for rendering the virtual space 70.

[0152] The rendering unit 325 according to this embodiment is capable of rendering an operation of the avatar 72 on the basis of the bone information included in the motion data of the actor 52 and the above avatar data.

[0153] Further, the rendering unit 325 according to this embodiment is capable of performing physical calculation based on the contact position and the hang time estimated on the basis of the period between the times of contact, estimating the trajectory of the virtual object 74, and rendering an operation of the virtual object 74 on the basis of the estimated trajectory and the above virtual object data.

[0154] Fig. 11 is a diagram describing estimation of the trajectory of the virtual object 74 in this embodiment.

[0155] Fig. 11 illustrates bone information Ma and a contact position at a time of contact Ta, bone information Mb and a contact position at a time of contact Tb, and bone information Mc and a contact position at a time of contact Tc.

[0156] In the case of rendering an operation of the virtual object 74 between the time of contact Ta and the time of contact Tb, the rendering unit 325 uses the coordinates (Ma's coordinates) of the contact part estimated on the basis of the bone information Ma as launch coordinates of the virtual object 74.

[0157] Further, the rendering unit 325 uses the coordinates (Mb's coordinates) of the contact part estimated on the basis of the bone information Mb as landing coordinates of the virtual object 74.

[0158] The rendering unit 325 is capable of estimating the trajectory of the virtual object 74 between the time of contact Ta and the time of contact Tb on the basis of the launch coordinates, the landing coordinates, and the hang time estimated from the time of contact Ta and the time of contact Tb.

[0159] Further, the rendering unit 325 is capable of rendering an operation of the virtual object 74 on the basis of the estimated trajectory of the virtual object 74 and the virtual object data.

[0160] Further, in the case of rendering an operation of the virtual object 74 between the time of contact Tb and the time of contact Tc, the rendering unit 325 uses the coordinates (Mb's coordinates) of the contact part estimated on the basis of the bone information Mb as launch coordinates of the virtual object 74.

[0161] Further, the rendering unit 325 uses the coordinates (Mc's coordinates) of the contact part estimated on the basis of the bone information Mc as landing coordinates of the virtual object 74.

[0162] The rendering unit 325 is capable of estimating the trajectory of the virtual object 74 between the time of contact Tb and the time of contact Tc on the basis of the launch coordinates, the landing coordinates, and the hang time estimated from the time of contact Tb and the time of contact Tc.

[0163] Further, the rendering unit 325 is capable of rendering an operation of the virtual object 74 on the basis of the estimated trajectory of the virtual object 74 and the virtual object data.

[0164] The estimation of the trajectory of the virtual object 74 in this embodiment has been described above. In accordance with the above processing, it is possible to reproduce an interaction between the actor 52 and the object 54 in the real space 50 as an interaction between the avatar 72 and the virtual object 74 in the virtual space 70.

[0165] <<1.4. Application example>> Next, an application example of the information processing method according to this embodiment will be described.

[0166] As described above, the information processing method according to this embodiment is appliable to reproduction of lifting in soccer or the like.

[0167] Fig. 12 is a flowchart showing an example of the flow of reproduction of lifting in this embodiment.

[0168] In the case of an example shown in Fig. 12, first, the time synchronization unit 322 executes time synchronization processing (S201).

[0169] After the time synchronization processing in Step S201, lifting measurement is started (S202).

[0170] The lifting may be started when the actor 52 (hereinafter, referred to as a user in this example) releases the object 54 (hereinafter, referred to as a ball in this example) from his / her hand.

[0171] After the lifting measurement is started in Step S202, first, the contact time detection unit 323 determines whether or not the object 54, i.e., the ball is positioned on the ground (S203).

[0172] The contact time detection unit 323 may estimate the approximate height of the ball on the basis of inertial navigation using the motion data acquired by the second sensor 20 attached to the ball, and determine, on the basis of the result of the estimation, whether or not the ball is positioned on the ground. The contact time detection unit 323 determines whether or not the ball is positioned on the ground on the basis of, for example, whether or not the approximate height of the ball is smaller than a predetermined height.

[0173] In the case where the contact time detection unit 323 determines that the ball is positioned on the ground (S203: YES), the lifting measurement ends (S209).

[0174] Meanwhile, in the case where it is determined that the ball is not positioned on the ground (S203: NO), the contact time detection unit 323 attempts to detect contact between the user and the ball (S204).

[0175] In the case where the contact between user and the ball has not been detected (S204: NO), the processing may return to Step S203.

[0176] Meanwhile, in the case where the contact between the user and the ball has been detected (S204: YES), the contact position estimation unit 324 estimates a contact position on the basis of the detected time of contact (S205).

[0177] The rendering unit 325 estimates the trajectory of the ball on the basis of the contact position estimated in Step S205, or the like (S206).

[0178] Subsequently, the rendering unit 325 executes rendering processing based on the estimated trajectory of the ball, or the like (S207).

[0179] The processing unit 320 attempts to detect an end operation such as retrieving the ball by the user with his / her hand (S208). In the case where an end operation has been detected (S208: YES), the processing unit 320 ends the lifting measurement (S209), and the series of processing also ends.

[0180] Meanwhile, in the case where an end operation has not been detected (S208: NO), the processing may return to Step S203.

[0181] The flow of reproduction of lifting in this embodiment has been described above.

[0182] As described above, in accordance with the information processing method according to this embodiment, it is possible to reproduce an interaction between the user and the ball in the real space 50, in the virtual space 70.

[0183] Note that the scope of application of the information processing method according to this embodiment is not limited to interactions such as lifting, where there is a one-to-one relationship between the actor 52 and the object 54.

[0184] For example, the information processing method according to this embodiment is also applicable to interactions in which the actors 52 and the object 54 are in an N-to-one relationship.

[0185] Fig. 13 is a diagram describing a case where the information processing method according to this embodiment is applied to reproduction of a volleyball match.

[0186] As shown in Fig. 13, a volleyball match is played by a plurality of actors 52 (players). Note that although four players of actors 52a to 52d are illustrated in Fig. 13, the number of actors 52 is not limited. The information processing method according to this embodiment is similarly appliable to reproduction of a 6-player or 9-player volleyball match.

[0187] The first sensor group 10N (illustration omitted) is attached to each of the plurality of actors 52, and motion data of each of the plurality of actors 52 is acquired.

[0188] Further, the second sensor 20 (illustration omitted) is attached to the object 54, and motion data of the object 54 is acquired.

[0189] The information processing apparatus 30 according to this embodiment estimates, on the basis of each piece of acquired motion data, a time of contact and a contact position between the object 54 and each of the plurality of actors 52.

[0190] Further, the information processing apparatus 30 according to this embodiment estimates the trajectory of the object 54 on the basis of the estimated time of contact and contact position.

[0191] In accordance with the above processing, it is possible to reproduce interactions between the object 54 and the plurality of actors 52 in the virtual space 70, such as a volleyball match.

[0192] Further, the information processing method according to this embodiment is also appliable to reproduction of an interaction between the actor 52 and the object 54 via a tool 56.

[0193] Fig. 14 is a diagram describing a case where the information processing method according to this embodiment is applied to reproduction of a table tennis match.

[0194] In a table tennis match, the actor 52 (player) interacts with the object 54 (ball) via the tool 56 (racket).

[0195] The first sensor group 10N (illustration omitted) is attached to each of the plurality of actors 52, and motion data of each of the plurality of actors 52 is acquired.

[0196] Further, the second sensor 20 (illustration omitted) is attached to the object 54, and motion data of the object 54 is acquired.

[0197] Further, the information processing apparatus 30 detects a time of contact between the object 54 and the tool 56.

[0198] Further, the information processing apparatus 30 detects a time of contact between the object 54 and a table 58.

[0199] The information processing apparatus 30 may detect a time of contact between the object 54 and the tool 56 and a time of contact between the object 54 and the table 58 on the basis of sensing data acquired by inertial measurement units attached to the tool 56 and the table 58.

[0200] For example, the information processing apparatus 30 may detect, as a time of contact between the object 54 and the tool 56, the time at which an impact exceeding a threshold value was detected in the sensing data acquired by the inertial measurement unit attached to the tool 56.

[0201] Similarly, the information processing apparatus 30 may detect, as a time of contact between the object 54 and the table 58, the time at which an impact exceeding a threshold value was detected in the sensing data acquired by the inertial measurement unit attached to the table 58.

[0202] Meanwhile, the information processing apparatus 30 may detect a time of contact between the object 54 and the tool 56 and a time of contact between the object 54 and the table 58 on the basis of the sensing data acquired by a microphone installed in the periphery of the table 58.

[0203] In this case, for example, the information processing apparatus 30 may detect, as a time of contact between the tool 56 or the table 58 and the object 54, the time at which a sound pressure exceeding a threshold value was detected.

[0204] In this case, for example, the information processing apparatus 30 may estimate a contact position on the basis of the motion data of the actor 52.

[0205] Further, the information processing apparatus 30 may estimate the position of a sound source on the basis of the sensing data acquired by a plurality of microphones, and estimate a target that has come into contact with the object 54 and the contact position.

[0206] The information processing apparatus 30 is capable of estimating the trajectory of the object 54 on the basis of the time of contact and contact position estimated as described above.

[0207] In accordance with the processing described above, for example, it is possible to reproduce an interaction between the actor 52 and the object 54 via the tool 56 in the virtual space 70, such as a table tennis match.

[0208] Further, in the case of reproducing an interaction between the actor 52 and the object 54 via the tool 56 in the virtual space 70, operation analysis based on the detected or estimated information may be performed, and the operation analysis result may be presented.

[0209] Fig. 15 is a diagram showing an example of presentation of an operation analysis result in this embodiment. Fig. 15 shows an example of presentation of an operation analysis result of a tennis swing.

[0210] The information processing apparatus 30 is capable of reproducing, by the processing described with reference to Fig. 14, the operations of the actor 52, the object 54, and the tool 56 in the real space 50 as operations of the avatar 72, the virtual object 74, and a virtual tool 76 in the virtual space 70.

[0211] Further, the information processing apparatus 30 may perform various types of operation analysis during the swing on the basis of the time of contact, the motion data of the actor 52, and the like, and present an analysis result 79 to a user.

[0212] As shown in Fig. 15, the analysis result 79 can include, for example, the hand angle, elbow angle, and swing speed.

[0213] As described above, in accordance with the information processing method according to this embodiment, it is possible to perform operation analysis based on the detected or estimated information and present the operation analysis result.

[0214] Note that the information processing method according to this embodiment is also appliable to, for example, operation analysis such as at what timing the racket and the ball are in contact with each other.

[0215] Next, a case where the information processing method according to this embodiment is applied to an interaction in which the actor 52 and the objects 54 are in a one-to-N relationship will be described.

[0216] Fig. 16 is a diagram describing a case where the information processing method according to this embodiment is applied to reproduction of juggling.

[0217] Fig. 16 illustrates the actor 52 performs juggling using a plurality of objects 54a to 54d.

[0218] The first sensor group 10N (illustration omitted) is attached to the actor 52, and motion data of the actor 52 is acquired.

[0219] Further, the second sensor 20 (illustration omitted) is attached to each of the plurality of objects 54, and motion data of each of the plurality of objects 54 is acquired.

[0220] The information processing apparatus 30 according to this embodiment estimates, on the basis of each piece of acquired motion data, a time of contact between each of the plurality of objects 54 and the actor 52, and the contact position.

[0221] Further, the information processing apparatus 30 according to this embodiment estimates the trajectory of each of the plurality of objects 54 on the basis of the estimated time of contact and contact position.

[0222] In accordance with the above processing, it is possible to reproduce interactions between the plurality of objects 54 and the actor 52 in the virtual space 70, such as juggling.

[0223] Next, rendering of an interaction, which is difficult in reality, in this embodiment will be described with reference to Fig. 17.

[0224] An application example in which lifting in the real space 50 is reproduced in the virtual space 70 has been described above. However, in reality, there is the actor 52 who cannot perform lifting.

[0225] For the actor 52 who cannot perform lifting, lifting is a difficult interaction in reality.

[0226] In accordance with the information processing method according to this embodiment, it is possible to reproduce such an interaction that is difficult in reality, in the virtual space 70 in a pseudo manner.

[0227] First, in the real space 50, the actor 52 who cannot perform lifting performs an operation that mimics lifting without the object 54.

[0228] The first sensor group 10N is attached to the actor 52, and motion data of the actor 52 is acquired.

[0229] In addition to the above motion data, the time of contact and contact position designated by a user are input to the rendering unit 325.

[0230] The time of contact may be a time at which the user desires the avatar 72 and the virtual object 74 to come into contact with each other, and the contact position may be a position at which the user desires the avatar 72 and the virtual object 74 to come into contact with each other.

[0231] The rendering unit 325 performs rendering of the avatar 72 and the virtual object 74 in the virtual space 70 on the basis of the input motion data, time of contact, and contact position.

[0232] In accordance with the above processing, it is possible to reproduce lifting that is difficult in reality by the actor 52, in the virtual space 70 in a pseudo manner, and provide a new user experience to the actor 52.

[0233] <2. Hardware configuration example> Next, a hardware configuration example of the information processing apparatus 30 according to an embodiment of the present disclosure will be described. Fig. 18 is a block diagram showing a hardware configuration example of an information processing apparatus 90 according to an embodiment of the present disclosure. The information processing apparatus 90 may be an apparatus having a hardware configuration equivalent to that of the information processing apparatus 30.

[0234] As shown in Fig. 18, the information processing apparatus 90 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is merely an example, and some components may be omitted. Further, the information processing apparatus 90 may further include a component other than the components shown here.

[0235] (Processor 871) The processor 871 functions as, for example, an arithmetic processing unit or control device, and controls the operation of each component in general or in part on the basis of various programs recorded on the ROM 872, the RAM 873, the storage 880, or a removable storage medium 901.

[0236] (ROM 872 and RAM 873) The ROM 872 is a means for storing a program to be loaded into the processor 871, data to be used for calculation, and the like. The RAM 873 temporarily or permanently stores, for example, a program to be loaded into the processor 871, various parameters that appropriately change when executing the program, and the like.

[0237] (Host bus 874, bridge 875, external bus 876, and interface 877) The processor 871, the ROM 872, and the RAM 873 are connected to each other via, for example, the host bus 874 capable of performing high-speed data transmission. Meanwhile, the host bus 874 is connected to the external bus 876 whose data transmission rate is relatively low, via the bridge 875, for example. Further, the external bus 876 is connected to various components via the interface 877.

[0238] (Input device 878) For example, a mouse, a keyboard, a touch panel, a button, a switch, or a lever is used as the input device 878. Further, a remote controller capable of transmitting a control signal using infrared rays or other radio waves is used as the input device 878 in some cases. Further, the input device 878 includes a voice input device such as a microphone.

[0239] (Output device 879) The output device 879 is a device capable of visually or audibly notifying a user of acquired information, e.g., a display device such as a cathode ray tube (CRT), an LCD, and an organic EL, an audio output device such as a speaker and a headphone, a printer, a mobile phone, or a facsimile machine. Further, the output device 879 according to an embodiment of the present disclosure includes various vibration devices capable of outputting a haptic stimulus.

[0240] (Storage 880) The storage 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device is used as the storage 880.

[0241] (Drive 881) The drive 881 is, for example, a device that reads information recorded on the removable storage medium 901 such as a magnetic disc, an optical disc, a magneto-optical disc, or a semiconductor memory or writes information to the removable storage medium 901.

[0242] (Removable storage medium 901) The removable storage medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, or various semiconductor storage media. It goes without saying that the removable storage medium 901 may be an IC card with a contactless IC chip, an electronic apparatus, or the like.

[0243] (Connection port 882) The connection port 882 is a port for connecting an external connection device 902, such as a universal serial bus (USB) port, an IEEE1394 port, a small computer system interface (SCSI), an RS-232C port, and an optical audio terminal.

[0244] (External connection device 902) The external connection device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0245] (Communication device 883) The communication device 883 is a communication device for connecting to a network, and is, for example, a wired or wireless LAN, a communication card for Bluetooth (registered trademark) or wireless USB (WUSB), a router for optical communication, a router for asymmetric digital subscriber line (ADSL), or a modem for various types of communication.

[0246] <3.Conclusion> As described above, the information processing apparatus 30 according to an embodiment of the present disclosure includes: the time synchronization unit 322 that synchronizes times of the motion data of the object 54 in the real space 50 and the motion data of the actor 52 in the real space 50; the contact time detection unit 323 that detects a time of contact between the object 54 and the actor 52 on the basis of the synchronized motion data of the object 54 and the synchronized motion data of the actor 52; and the contact position estimation unit 324 that estimates a contact position between the object 54 and the actor 52 on the basis of the synchronized motion data of the object 54, the synchronized motion data of the actor 52, and the time of contact.

[0247] In accordance with the above configuration, it is possible to reproduce an interaction between an actor and an object with a simpler configuration.

[0248] Although a suitable embodiment of the present disclosure has been described above in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such an example. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications and alterations within the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0249] Further, the steps of the processing described in the present disclosure do not necessarily need to be processed in chronological order in accordance with the order described in the flowchart or sequence diagram. For example, the steps of processing executed by each device may be processed in an order different from the described order or may be processed in parallel.

[0250] Further, the series of processing executed by each device described in the present disclosure may be realized by programs stored in a non-transitory computer readable storage medium. Each program is loaded into, for example, a RAM when executed by a computer, and executed by a processor such as a CPU. The storage medium is, for example, a magnetic disc, an optical disc, a magneto-optical disc, or a flash memory. Further, the program may be distributed via, for example, a network without using a storage medium.

[0251] Further, the effects described in the present specification are merely descriptive or exemplary and are not limitative. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification, in addition to or instead of the above effects.

[0252] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing system including: circuitry configured to acquire sensor data indicating motion data of at least one actor and motion data of at least one object, synchronize the motion data of the at least one actor and the motion data of the at least one object, detect a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object, and estimate a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact. (2) The information processing system according to (1), in which the sensor data includes first sensor data acquired from at least one first sensor attached to the at least one actor and second sensor data acquired from at least one second sensor attached to the at least one object. (3) The information processing system according to (1) or (2), in which the circuitry configured to synchronize the motion data of the at least one actor and the motion data of the at least one object at a time when the sensor data indicates an impact exceeding a threshold value. (4) The information processing system according to any one of (1) to (3), in which the impact exceeding the threshold value is detected based on an acceleration of the at least one object indicated by the sensor data. (5) The information processing system according to any one of (1) to (4), in which the at least one first sensor includes an acceleration sensor and a gyro sensor, and in which the motion data of at least one actor indicated by the first sensor data includes acceleration and velocity. (6) The information processing system according to any one of (1) to (5), in which the at least one first sensor includes a plurality of first sensors attached to a plurality of attachment positions on each actor. (7) The information processing system according to any one of (1) to (6), in which each first sensor includes an inertial measurement unit (IMU). (8) The information processing system according to any one of (1) to (7), in which the first sensor data acquired from the plurality of first sensors is used to determine joint information of the at least one actor. (9) The information processing system according to any one of (1) to (8), in which the joint information includes three-dimensional coordinates of a plurality of joints of each actor estimated from an acceleration and a velocity indicated by the first sensor data acquired by the plurality of first sensors. (10) The information processing system according to any one of (1) to (9), in which the circuitry configured to estimate the contact position based on the three-dimensional coordinates of each joint at the time of each contact. (11) The information processing system according to any one of (1) to (10), in which the circuitry configured to estimate the contact position using a contact position classifier generated by machine learning based on the three-dimensional coordinates of each joint at the time of each contact. (12) The information processing system according to any one of (1) to (11), in which the at least one second sensor includes an acceleration sensor and a gyro sensor. (13) The information processing system according to any one of (1) to (12), in which the circuitry is further configured to generate at least one avatar corresponding to the motion data of the at least one actor and at least one virtual object corresponding to the motion data of the at least one object. (14) The information processing system according to any one of (1) to (13), in which the circuitry is further configured to initiate display of the at least one avatar and the at least one virtual object in a virtual space. (15) The information processing system according to any one of (1) to (14), in which the circuitry is further configured to initiate display of an interaction between the at least one avatar and the at least one virtual object in the virtual space based on the estimated contact position between the at least one actor and the at least one object at each time of contact. (16) The information processing system according to any one of (1) to (15), in which the interaction between the at least one avatar and the at least one virtual object in the virtual space includes a period between detected times of contact. (17) The information processing system according to any one of (1) to (16), in which the interaction including the period between the detected times of contact is determined based on a trajectory of the at least one object estimated based on each contact. (18) The information processing system according to any one of (1) to (17), in which the trajectory is estimated based on launch coordinates of the estimated contact position at the time of a first contact, landing coordinates at the time of a second contact, and a hang time between the time of the first contact and the time of the second contact. (19) An information processing method including: acquiring sensor data indicating motion data of at least one actor and motion data of at least one object; synchronizing the motion data of the at least one actor and the motion data of the at least one object; detecting a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object; and estimating a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact. (20) A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method including: acquiring sensor data indicating motion data of at least one actor and motion data of at least one object, synchronizing the motion data of the at least one actor and the motion data of the at least one object, detecting a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object, and estimating a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

[0253] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing apparatus, including: a time synchronization unit that synchronizes times of motion data of an object in a real space and motion data of an actor in the real place; a contact time detection unit that detects a time of contact between the object and the actor on the basis of the synchronized motion data of the object and the synchronized motion data of the actor; and a contact position estimation unit that estimates a contact position between the object and the actor on the basis of the synchronized motion data of the object, the synchronized motion data of the actor, and the time of contact. (2) The information processing apparatus according to (1) above, further including a rendering unit that renders, on the basis of the time of contact between and the contact position, an interaction between an avatar corresponding to the actor in a virtual space and a virtual object corresponding to the object in the virtual space. (3) The information processing apparatus according to (1) or (2) above, in which the contact time detection unit detects, on the basis of the motion data of the object, a time at which an impact exceeding a threshold value occurred on the object, as the time of contact. (4) The information processing apparatus according to any one of (1) to (3) above, in which the contact position estimation unit estimates the contact position on the basis of bone information of the actor included in the motion data of the actor. (5) The information processing apparatus according to (4) above, in which the bone information includes coordinates of a plurality of joints of the actor. (6) The information processing apparatus according to (5) above, in which the contact position estimation unit estimates the contact position on the basis of a positional relationship between the plurality of joints at the time of contact. (7) The information processing apparatus according to (6) above, in which the contact position estimation unit estimates the contact position using a rule-based method. (8) The information processing apparatus according to (6) above, in which the contact position estimation unit estimates the contact position using a machine learning method. (9) The information processing apparatus according to any one of (4) to (8) above, in which the contact position estimation unit estimates a body part of the actor as the contact position and outputs information regarding the estimated body part. (10) The information processing apparatus according to (2) above, in which the rendering unit performs physical calculation based on the contact position and a hang time estimated on the basis of a period between times of contact to render a trajectory of the virtual object. (11) The information processing apparatus according to (2) above, in which the rendering unit renders an operation of the avatar on the basis of bone information of the actor included in the motion data of the actor. (12) The information processing apparatus according to any one of (1) to (11) above, in which the time synchronization unit synchronizes a time at which an impact exceeding a threshold value was detected on the object and a time at which re-contact of the actor and a ground was detected. (13) The information processing apparatus according to any one of (1) to (12) above, in which the contact position estimation unit estimates a contact position between the object and each of a plurality of actors. (14) The information processing apparatus according to any one of (1) to (13) above, in which the contact position estimation unit estimates a contact position between each of a plurality of objects and the actor. (15) The information processing apparatus according to any one of (1) to (14) above, in which the motion data of the actor is acquired by a plurality of inertial measurement units attached to the actor. (16) The information processing apparatus according to any one of (1) to (15) above, in which the motion data of the object is acquired by an inertial measurement unit attached to the object. (17) An information processing method, including: by a processor, synchronizing times of motion data of an object in a real space and motion data of an actor in the real place; detecting a time of contact between the object and the actor on the basis of the synchronized motion data of the object and the synchronized motion data of the actor; and estimating a contact position between the object and the actor on the basis of the synchronized motion data of the object, the synchronized motion data of the actor, and the time of contact. (18) A program that causes a computer to function as an information processing apparatus that includes a time synchronization unit that synchronizes times of motion data of an object in a real space and motion data of an actor in the real place, a contact time detection unit that detects a time of contact between the object and the actor on the basis of the synchronized motion data of the object and the synchronized motion data of the actor, and a contact position estimation unit that estimates a contact position between the object and the actor on the basis of the synchronized motion data of the object, the synchronized motion data of the actor, and the time of contact.

[0254] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0255] 10 first sensor 10N first sensor group 20 second sensor 30 information processing apparatus 320 processing unit 321 calibration unit 322 time synchronization unit 323 contact time detection unit 324 contact position estimation unit 325 rendering unit 326 contact position classifier 50 real space 52 actor 54 object 55 tool 72 avatar 74 virtual object 76 virtual tool

Claims

1. An information processing system comprising: circuitry configured to acquire sensor data indicating motion data of at least one actor and motion data of at least one object, synchronize the motion data of the at least one actor and the motion data of the at least one object, detect a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object, and estimate a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

2. The information processing system according to claim 1, wherein the sensor data includes first sensor data acquired from at least one first sensor attached to the at least one actor and second sensor data acquired from at least one second sensor attached to the at least one object.

3. The information processing system according to claim 2, wherein the circuitry is configured to synchronize the motion data of the at least one actor and the motion data of the at least one object at a time when the sensor data indicates an impact exceeding a threshold value.

4. The information processing system according to claim 3, wherein the impact exceeding the threshold value is detected based on an acceleration of the at least one object indicated by the sensor data.

5. The information processing system according to claim 2, wherein the at least one first sensor includes an acceleration sensor and a gyro sensor, and wherein the motion data of at least one actor indicated by the first sensor data includes acceleration and velocity.

6. The information processing system according to claim 2, wherein the at least one first sensor includes a plurality of first sensors attached to a plurality of attachment positions on each actor.

7. The information processing system according to claim 6, wherein each first sensor includes an inertial measurement unit (IMU).

8. The information processing system according to claim 6, wherein the first sensor data acquired from the plurality of first sensors is used to determine joint information of the at least one actor.

9. The information processing system according to claim 8, wherein the joint information includes three-dimensional coordinates of a plurality of joints of each actor estimated from an acceleration and a velocity indicated by the first sensor data acquired by the plurality of first sensors.

10. The information processing system according to claim 9, wherein the circuitry is configured to estimate the contact position based on the three-dimensional coordinates of each joint at the time of each contact.

11. The information processing system according to claim 10, wherein the circuitry is configured to estimate the contact position using a contact position classifier generated by machine learning based on the three-dimensional coordinates of each joint at the time of each contact.

12. The information processing system according to claim 2, wherein the at least one second sensor includes an acceleration sensor and a gyro sensor.

13. The information processing system according to claim 1, wherein the circuitry is further configured to generate at least one avatar corresponding to the motion data of the at least one actor and at least one virtual object corresponding to the motion data of the at least one object.

14. The information processing system according to claim 13, wherein the circuitry is further configured to initiate display of the at least one avatar and the at least one virtual object in a virtual space.

15. The information processing system according to claim 14, wherein the circuitry is further configured to initiate display of an interaction between the at least one avatar and the at least one virtual object in the virtual space based on the estimated contact position between the at least one actor and the at least one object at each time of contact.

16. The information processing system according to claim 15, wherein the interaction between the at least one avatar and the at least one virtual object in the virtual space includes a period between detected times of contact.

17. The information processing system according to claim 16, wherein the interaction including the period between the detected times of contact is determined based on a trajectory of the at least one object estimated based on each contact.

18. The information processing system according to claim 17, wherein the trajectory is estimated based on launch coordinates of the estimated contact position at the time of a first contact, landing coordinates at the time of a second contact, and a hang time between the time of the first contact and the time of the second contact.

19. An information processing method comprising: acquiring sensor data indicating motion data of at least one actor and motion data of at least one object; synchronizing the motion data of the at least one actor and the motion data of the at least one object; detecting a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object; and estimating a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

20. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to function as execute an information processing method, the method comprising: acquiring sensor data indicating motion data of at least one actor and motion data of at least one object; synchronizing the motion data of the at least one actor and the motion data of the at least one object; detecting a time of each contact between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object; and estimating a contact position between the at least one actor and the at least one object based on the synchronized motion data of the at least one actor and the synchronized motion data of the at least one object at the time of each contact.

Images