Information processing apparatus and information processing method

Abstract

An information processing apparatus according to an embodiment includes a processing unit that measures positions of a plurality of parts of a user based on respective positions of a plurality of measurement devices each mounted to the user, and acquires image data of the user, and the image data is acquired from the plurality of measurement devices, respectively, the position of each of the plurality of measuring devices is estimated by the measuring device itself or measured using another measuring device installed to the user.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and an information processing method. Background Art

[0002] In recent years, techniques for tracking the posture and movement of a part or all of a human body and reflecting the posture and movement on an avatar in a virtual space have been developed. For example, the technique described below has been proposed in Non-Patent Document 1: With the relative positions of cameras fixed, triangulation is performed on the body parts of a user based on image data of the user respectively captured by two cameras attached to the user, thereby measuring the positions of the body parts.

[0003] Citation List

[0004] Non-Patent Document

[0005] Non-Patent Document 1: Helge Rhodin, et al., EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras, (US), 2016, ACM Transactions on Graphics, Volume 35, Issue 6. Summary of the Invention

[0006] Technical Problem

[0007] In the technique disclosed in Non-Patent Document 1, an optical device is used and a plurality of measurement devices designed to be attached to a user are fixed such that the distance between the measurement devices is always a predetermined distance and the measurement devices protrude from the user. On the other hand, there is no specific examination regarding the use of a plurality of measurement devices to measure the positions of a user's body parts, the plurality of measurement devices having a form that enables the distance between the plurality of measurement devices using the optical device to be flexibly changed, and the plurality of measurement devices being attached in a form that is in close contact with or close to the user's body.

[0008] One aspect of the present disclosure is to be able to measure the position of a user's body part even when the positions of the measurement devices move freely.

[0009] Solution to the Problem

[0010] An information processing apparatus according to an aspect of the present disclosure includes: a processing unit configured to measure positions of a plurality of parts of a user based on positions of a plurality of measurement devices respectively attached to the user, to obtain image data of the user and image data obtained by each of the plurality of measurement devices, wherein the position of each measurement device of the plurality of measurement devices is self-estimated by the measurement device or measured using another measurement device attached to the user.

[0011] An information processing method according to an aspect of the present disclosure includes: measuring positions of a plurality of parts of a user based on positions of a plurality of measurement devices respectively attached to the user, to obtain image data of the user and image data obtained by each of the plurality of measurement devices, wherein the position of each measurement device of the plurality of measurement devices is self-estimated by the measurement device or measured using another measurement device attached to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram showing an exemplary configuration of a human body tracking system according to a first embodiment.

[0013] Figure 2 is a block diagram showing an exemplary functional configuration of each device constituting the human body tracking system shown in the first Figure 1 in which.

[0014] Figure 3 is a diagram showing an example of a video displayed on a display device in a human body tracking system according to a first embodiment.

[0015] Figure 4 is an external view showing an exemplary schematic configuration of a measurement device according to a first example of a first embodiment.

[0016] Figure 5 is an external view showing an exemplary schematic configuration of a measurement device according to a second example of a first embodiment.

[0017] Figure 6 is an external view showing an exemplary schematic configuration of a measurement device according to a third example of a first embodiment.

[0018] Figure 7 is a diagram for showing a method of measuring a wrist position according to a first embodiment.

[0019] Figure 8 is a diagram showing a method of measuring positions of an elbow and a shoulder according to a first embodiment.

[0020] Fig. 9 is a schematic diagram for showing a method of measuring positions of a knee and an ankle according to a first embodiment.

[0021] Fig.10 It is a diagram for explaining position measurement using triangulation.

[0022] Fig.11 It is a diagram showing a method for measuring the positions of the measurement head and the main body according to the first embodiment.

[0023] Fig.12 It is a diagram showing position measurement using a distance measurement sensor.

[0024] Fig.13 It is a diagram (part 1) for showing a method for measuring the shape of the hand / fingers according to the first embodiment.

[0025] Fig.14 It is a diagram (part 2) for showing a method for measuring the shape of the hand / fingers according to the first embodiment.

[0026] Fig.15 It is a diagram (part 3) for showing a method for measuring the shape of the hand / fingers according to the first embodiment.

[0027] Fig.16 It is a diagram (part 4) for showing a method for measuring the shape of the hand / fingers according to the first embodiment.

[0028] Fig.17 It is an external view showing an example of a measuring device for measuring the shape of the hand / fingers according to the first embodiment.

[0029] Fig.18 It is a flowchart showing a schematic operation example of coordinate system integration processing according to the first embodiment.

[0030] Fig.19 It is a diagram showing an example of the space where a user wearing the measuring device exists in the description of the first embodiment.

[0031] Fig. 20 It is showing Fig.19 A diagram showing an example of a set of coordinate systems of the space shown in.

[0032] Fig.21 It is a flowchart showing a schematic operation example of a human body tracking system according to the first embodiment.

[0033] Fig. 22 It is a flowchart showing an example of the process of map integration processing according to the first embodiment.

[0034] Fig.23 It is for supplementing Fig. 22 A diagram (part 1) of the process shown in.

[0035] Fig.24 is a diagram (part 2) for supplementing the process shown in Fig. 22 .

[0036] Fig.25 is a diagram (part 3) for supplementing the process shown in Fig. 22 .

[0037] Fig.26 is a diagram showing the map integration of the first technique according to the first embodiment.

[0038] Fig. 27 is a diagram for showing another map integration of the first technique of the first embodiment.

[0039] Fig.28 is a flowchart showing an example of the process of map integration processing of a modification of the first technique according to the first embodiment.

[0040] Fig.29 is a diagram (part 1) for showing the map integration of the second technique of the first embodiment.

[0041] Fig.30 is a diagram (part 2) for showing the map integration of the second technique of the first embodiment.

[0042] Fig.31 is a diagram (part 1) showing the map integration of the first modification of the second technique according to the first embodiment.

[0043] Fig.32 is a diagram (part 2) showing the map integration of the first modification of the second technique according to the first embodiment.

[0044] Fig.33 is a diagram showing the map integration of the second modification of the second technique according to the first embodiment.

[0045] Fig.34 is a diagram showing the map integration of the third technique according to the first embodiment.

[0046] Fig.35 is a flowchart for explaining the process of SLAM processing according to the first example of the first embodiment.

[0047] Fig.36 is a flowchart for explaining the process of SLAM processing according to the second example of the first embodiment.

[0048] Fig.37 is a flowchart for explaining the process of SLAM processing according to the third example of the first embodiment.

[0049] Fig.38 It is a diagram (part 1) showing a list of processes according to the first embodiment.

[0050] Fig.39 It is a diagram (part 1) showing a list of output data according to the first embodiment.

[0051] Fig.40 It is a schematic diagram (part 1) showing a list of combined variant examples of units that execute respective processes according to the first embodiment.

[0052] Fig.41 It is a diagram (part 2) showing a list of processes according to the first embodiment.

[0053] Fig.42 It is a diagram (part 2) showing a list of output data according to the first embodiment.

[0054] Fig.43 It is a schematic diagram (part 2) showing a list of combined variant examples of units that execute respective processes according to the first embodiment.

[0055] Fig.44 It is a diagram (part 3) illustrating a list of processes according to the first embodiment.

[0056] Fig.45 It is a diagram (part 3) showing a list of output data according to the first embodiment.

[0057] Fig.46 It is a diagram (part 3) illustrating a list of combined variant examples of units that execute respective processes according to the first embodiment.

[0058] Fig.47 It is a diagram (part 4) illustrating a list of processes according to the first embodiment.

[0059] Fig.48 It is a diagram (part 4) showing a list of output data according to the first embodiment.

[0060] Fig.49 It is a diagram (part 4) illustrating a list of combined variant examples of units that execute respective processes according to the first embodiment.

[0061] Fig.50 It is a schematic diagram showing a schematic configuration example of a human tracking system according to the first modification of the first embodiment.

[0062] Fig.51 It is a schematic diagram showing a schematic configuration example of a human tracking system according to the second modification of the first embodiment.

[0063] Fig.52It is a schematic diagram showing a schematic configuration example of a human body tracking system according to a third modification of the first embodiment.

[0064] Fig.53 It is a schematic diagram showing a schematic configuration example of a human body tracking system according to a fourth modification of the first embodiment.

[0065] Fig.54 It is a schematic diagram showing a schematic configuration example of a human body tracking system according to a fifth modification of the first embodiment.

[0066] Fig.55 It is a schematic diagram showing a schematic configuration example of a human body tracking system according to a sixth modification of the first embodiment.

[0067] Fig.56 It is a schematic diagram showing a schematic configuration example of a human body tracking system according to the second embodiment.

[0068] Fig.57 It is a diagram for showing the estimation of the position and orientation of a measuring device.

[0069] Fig.58 It is a flowchart showing an example of a process for estimating the position and orientation of a measuring device.

[0070] Fig.59 It is a diagram illustrating an example of an apron attached to a user.

[0071] Fig.60 It is a diagram showing an example of a set of world coordinate systems of a reference apron according to the second embodiment.

[0072] Fig.61 It is a diagram for showing the integration of coordinates.

[0073] Fig.62 It is a schematic diagram showing a schematic configuration example of a human body tracking system.

[0074] Fig.63 It is a sketch for showing a transformation process according to the first example of the second embodiment.

[0075] Fig.64 It is a sketch for showing a transformation process according to the second example of the second embodiment.

[0076] Fig.65 It is a sketch for showing a transformation process according to the third example of the second embodiment.

[0077] Fig.66 It is a sketch (part 1) showing a processing list according to the second embodiment.

[0078] Fig.67 It is a diagram (part 1) showing a list of output data according to the second embodiment.

[0079] Fig.68 It is a diagram (part 1) exemplifying a list of combined modification examples of units that execute respective processes according to the second embodiment.

[0080] Fig.69 It is a schematic diagram (part 2) showing a list of processes according to the second embodiment.

[0081] Fig.70 It is a diagram (part 2) showing a list of output data according to the second embodiment.

[0082] Fig.71 It is a diagram (part 2) exemplifying a list of combined modification examples of units that execute respective processes according to the second embodiment.

[0083] Fig.72 It is a schematic diagram (part 3) showing a list of processes according to the second embodiment.

[0084] Fig.73 It is a diagram (part 3) showing a list of output data according to the second embodiment.

[0085] Fig.74 It is a diagram (part 3) exemplifying a list of combined modification examples of units that execute respective processes according to the second embodiment.

[0086] Fig.75 It is a schematic diagram (part 4) showing a list of processes according to the second embodiment.

[0087] Fig.76 It is a diagram (part 4) showing a list of output data according to the second embodiment.

[0088] Fig.77 It is a schematic diagram (part 4) exemplifying a list of combined modification examples of units that execute respective processes according to the second embodiment.

[0089] Fig.78 It is a schematic diagram showing a schematic configuration example of a human tracking system according to the first modification example of the second embodiment.

[0090] Fig.79 It is a schematic diagram showing a schematic configuration example of a human tracking system according to the first modification example of the second embodiment.

[0091] Fig.80 It is a block diagram showing a schematic configuration example of a relay device according to the second modification example of the second embodiment.

[0092] Fig.81It is a schematic diagram showing a schematic configuration example of a human body tracking system as an information processing system according to the third embodiment.

[0093] Fig.82 It is a block diagram showing a schematic configuration example of a measurement device attached to the head according to the third embodiment.

[0094] Fig.83 It is a schematic diagram showing a schematic configuration example of a human body tracking system according to a modification of the third embodiment.

[0095] Fig.84 It is an external view showing a schematic configuration example of a measurement device attached to the user's head according to the third embodiment.

[0096] Fig.85 It is a sketch (part 1) showing an example of the arrangement of sensors and the FOV of each sensor in a measurement device attached to the user's head according to the third embodiment.

[0097] Fig.86 It is a sketch (part 2) showing an example of the arrangement of sensors and the FOV of each sensor in a measurement device attached to the user's head according to the third embodiment.

[0098] Fig.87 It is a sketch (part 3) showing an example of the arrangement of sensors and the FOV of each sensor in a measurement device attached to the user's head according to the third embodiment.

[0099] Fig.88 It is shown in Fig.74 Or a view showing an example of the arrangement and FOV of sensors in a side view of the reference measurement device shown in 71.

[0100] Fig.89 It is a diagram for showing a method of estimating the head position and wrist position according to the third embodiment.

[0101] Fig.90 It is a sketch for showing a method of estimating the positions of the knees, ankles, and torso according to the third embodiment.

[0102] Fig.91 It is a schematic diagram for showing the integration of coordinate systems according to the third embodiment.

[0103] Fig.92 It is a diagram (part 1) showing a list of processes according to the third embodiment.

[0104] Fig.93 It is a diagram (part 1) showing a list of output data according to the third embodiment.

[0105] Fig.94 It is a diagram (part 1) showing a list of combinations of units that execute each process according to the third embodiment.

[0106] Fig.95 It is a diagram (part 2) showing a list of processes according to the third embodiment.

[0107] Fig.96 It is a diagram (part 2) showing a list of output data according to the third embodiment.

[0108] Fig.97 It is a schematic diagram (part 2) showing a list of combination variants of units that execute each process according to the third embodiment.

[0109] Fig.98 It is a diagram (part 3) showing a list of processes according to the third embodiment.

[0110] Fig.99 It is a diagram (part 3) showing a list of output data according to the third embodiment.

[0111] Fig.100 It is a diagram (part 3) showing a list of combinations of units that execute each process according to the third embodiment.

[0112] Fig.101 It is a schematic diagram showing a schematic configuration example of an online conferencing system according to the first application embodiment of the present disclosure.

[0113] Fig.102 It is a flowchart showing a schematic operation example of a body activity posture observation system according to the second application embodiment of the present disclosure.

[0114] Fig.103 It is a processing flowchart of the first embodiment of the third application embodiment of the present application.

[0115] Fig.104 It is a processing flowchart of the second embodiment of the third application embodiment of the present application.

[0116] Fig.105 It is a hardware configuration diagram showing an embodiment of an information processing device that executes various types of processes according to the present disclosure. Detailed Description of the Embodiment

[0117] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following embodiments, the same reference numerals are assigned to the same parts and repeated descriptions are omitted.

[0118] The present disclosure will be described in the following order.

[0119] 0. Introduction

[0120] 1. First Embodiment

[0121] 1.1 System Configuration Example

[0122] 1.2 Functional Configuration Example

[0123] 1.3 Schematic Configuration Example of Measuring Device

[0124] 1.3.1 First Example

[0125] 1.3.2 Second Example

[0126] 1.3.3 Third Example

[0127] 1.4 Position Measurement Method for Each Part

[0128] 1.4.1 Wrist

[0129] 1.4.2 Elbow and Shoulder

[0130] 1.4.3 Knee and Ankle

[0131] 1.4.4 Head and Trunk

[0132] 1.4.5 Another Method for Measuring the Position of Each Site

[0133] 1.4.6 Method for Measuring the Shape of Hand / Finger

[0134] 1.5 Coordinate System

[0135] 1.6 Schematic Operation Example

[0136] 1.6.1 Flowchart

[0137] 1.6.2 Variation of the Method for Obtaining the Initial 3D Map

[0138] 1.6.3 Specific Technology for Map Integration

[0139] 1.6.4 Variation of the SLAM Execution Mode

[0140] 1.6.5 Method of Inputting the Measured Position of Each Part into the Application and Operating the Human Model 1.7 Combinatorial Variation of the Units Executing Each Process

[0141] 1.8 Variation Example of System Configuration

[0142] 1.8.1 First Variation Example

[0143] 1.8.2 Second Variation Example

[0144] 1.8.3 Third Variation Example

[0145] 1.8.4 Fourth Modified Example

[0146] 1.8.5 Fifth Modified Example

[0147] 1.8.6 Sixth Modified Example

[0148] 1.9 Overview

[0149] 2. Second Embodiment

[0150] 2.1 System Configuration Example

[0151] 2.2 Position Measurement Method for Each Part

[0152] 2.2.1 Wrist

[0153] 2.2.2 Elbow and Shoulder

[0154] 2.2.3 Knee and Ankle

[0155] 2.2.4 Head and Trunk

[0156] 2.2.5 Another Method for Measuring the Position of Each Site

[0157] 2.2.6 Method for Measuring the Shape of Hand / Finger

[0158] 2.3 Coordinate System

[0159] 2.4 Combined Deformation of Units for Executing Each Process

[0160] 2.5 Deformation of System Configuration

[0161] 2.5.1 First Modified Example

[0162] 2.5.2 Second Modified Example

[0163] 2.6 Overview

[0164] 3. Third Embodiment

[0165] 3.1 System Configuration Example

[0166] 3.1.1 Deformation of System Configuration

[0167] 3.2 Configuration Example of Measurement Device Attached to Head

[0168] 3.3 Position Measurement Method for Each Part

[0169] 3.3.1 Head and Wrist

[0170] 3.3.2 Elbow and Shoulder

[0171] 3.3.3 Knee, Ankle, and Trunk

[0172] 3.3.4 Another method for measuring the position of each site

[0173] 3.3.5 Method for measuring the shape of the hand / finger

[0174] 3.4 Coordinate system

[0175] 3.5 Combined deformation of units for performing respective processes

[0176] 3.6 Overview

[0177] 4. Application examples

[0178] 4.1 First application example

[0179] 4.2 Second application example

[0180] 4.3 Third application example

[0181] 5. Hardware configuration

[0182] 6. Conclusion

[0183] 0. Introduction

[0184] For example, in order to measure the position of a user's (object's) body part in body tracking (which can be interpreted as including posture and movement), Non-Patent Document 1 uses two measurement devices attached to the user in a state where the positional relationship between the two measurement devices is constantly fixed. Non-Patent Document 1 does not assume free movement of the positions of the measurement devices. Since the two measurement devices are devices with their positional relationship constantly fixed, the device disclosed in Non-Patent Document 1 has a large size and a structure that protrudes significantly from the user's body.

[0185] As a technology different from Non-Patent Document 1, there is also a technology of mounting inertial sensors on moving body parts. In this case, it is necessary to attach inertial sensors (inertial measurement units (IMUs)) to each of N body parts (N is a natural number representing the number of body parts for which position estimation is to be performed). For a user, a large number of sensors make the user feel uncomfortable when wearing the device and hinder the movement of the body, resulting in a possibility of failure to detect natural postures or movements.

[0186] At least some of the above problems will be addressed by the disclosed technology. For example, two measurement devices for collecting image data of a user are attached to the user and can move freely in position. The position of each measurement device is estimated or measured (undergoes estimation, etc.) according to the principles described below. The position of a body part of the user is measured based on the estimated position of each measurement device and the image data acquired by each measurement device. The target body part of the user is not restricted as long as it is a part captured in the image of the user's body imaged by each measurement device, and can be any part. In addition, the number of the user's target body parts is not restricted, and any number of parts can be measured. Naturally, regarding the number of the user's target body parts, the position of more body parts can be measured than the number of measurement devices. Since the positions of a large number of body parts of the user can be measured by a small number of measurement devices, the technology of the present disclosure can suppress the problems of discomfort during wearing and hindrance to user movement occurring in the user, which have traditionally been caused by attaching a large number of devices or a large number of devices to the user.

[0187] 1. First Embodiment

[0188] First, the first embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. The user of the human body tracking system 1 is referred to as user U in the drawings. The human body tracking system 1 illustrated in this embodiment can be applied to various systems that reflect the posture and movement of user U to a person or an avatar existing in a virtual space (including a virtual reproduction of the real space), for example, in real time or after the measurement is completed. Similar applications are applicable to the modifications and other embodiments described below.

[0189] 1.1 System Configuration Embodiment

[0190] Figure 1 is a schematic diagram showing a schematic configuration embodiment of a human body tracking system as an information processing system according to the present embodiment. This embodiment will describe a so-called human body tracking system that tracks the whole body of user U and reflects the posture and movement of user U to an avatar in a virtual space. However, the present disclosure is not limited thereto, and various modifications can be made, such as a system that tracks a part or all of the human body, or other objects such as animals or robots, and reflects the tracked body to an avatar in a virtual space.

[0191] As Figure 1As shown, the human body tracking system 1 includes, as devices attached to the user U: a plurality (two in this example) of measurement devices, namely the measurement device 10AL and the measurement device 10AR (hereinafter, when not distinguishing individual measurement devices, the reference numeral is described as "10"); and a relay device 20. In the present embodiment, it is assumed that, for example, the measurement device 10AL is attached to the left wrist of the user U and the measurement device 10AR is attached to the right wrist of the user U. The measurement device 10 may also be referred to as a client device, a measurement unit, etc.

[0192] In addition, the human body tracking system 1 includes: a communication device 30 that sends data to and receives data from the measurement device 10 via the relay device 20; an information processing device 40 (hereinafter also referred to as a server) that detects the posture and movement (hereinafter also referred to as posture, etc.) of the user U based on the data received by the communication device 30 and reflects the detected posture, etc. onto an avatar in a virtual space; and a display device 50 that displays images and videos generated by the information processing device 40. However, the relay device 20 and the communication device 30 can be appropriately omitted. In addition, the display device 50 is not an essential component of the human body tracking system 1 and can be omitted.

[0193] 1.2 Functional configuration embodiment

[0194] Figure 2 is a block diagram showing Figure 1 the functional configuration embodiment of each device constituting the human body tracking system shown in

[0195] (Measurement device 10)

[0196] As Figure 2 shown, each measurement device 10 includes: a plurality of sensors 111-1 to 111-n (hereinafter, when not distinguishing individual sensors, the reference numeral is described as "111"); an IMU 112; a processing unit 113; a recording unit 114; a communication unit 115; and a power supply 116. For example, the sensors 111, the IMU 112, the processing unit 113, the recording unit 114, and the communication unit 115 are interconnected to enable mutual data transmission and reception via an internal bus.

[0197] The power supply 116 supplies power to each unit in the measurement device 10. The power supply 116 may include a rechargeable secondary battery or a primary battery such as a dry battery.

[0198] Each sensor 111 can be various sensors capable of acquiring information such as an image of an object present within a viewing angle (hereinafter, also referred to as imaging) and distance information. Examples of such sensors include: an image sensor that acquires a color image or a monochrome image (including an IR sensor that acquires an infrared (IR) image); a distance measurement sensor that acquires the distance to an object; an event-based vision sensor (EVS) that detects a change in luminance as an event; and a hybrid sensor having the functions of two or more of these sensors. For example, the EVS has a higher frame rate than an image sensor or a distance measurement sensor, and is therefore preferably used as the sensor 111 for measuring a part that moves at high speed.

[0199] In the case of the IR sensor, the sensor 111 may include a light source capable of emitting IR light toward an object. The distance measurement sensor may also be a time-of-flight (ToF) sensor (such as LiDAR), various distance measurement sensors (such as RADAR and ultrasonic sensors). In the case of the ToF sensor, the distance measurement sensor may use a direct ToF method or an indirect ToF method. Hereinafter, for simplicity, the image data, depth data, event data, etc. acquired by the sensor 111 including an image sensor, a distance measurement sensor, an EVS, etc. are also collectively referred to as image data.

[0200] The sensor group including a plurality of sensors 111 in each of the measurement devices 10AL and 10AR is a detector for estimating the position of each body part of the user U (e.g., left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, head, etc.). Therefore, the sensor group of the measurement device 10AL and the sensor group of the measurement device 10AR preferably have a field of view range capable of imaging the entire body of the user U as much as possible, regardless of the position and posture of the wrists of the user U wearing each of the measurement devices 10AL and 10AR.

[0201] Hereinafter, the body part of the user U may be simply referred to as a part. The measurement of a part can be interpreted as obtaining a part, for example, estimating, identifying, measuring, etc., and these can be appropriately paraphrased within a range without causing contradictions.

[0202] The IMU 112 detects movement in six degrees of freedom (6DoF) directions (i.e., the X, Y, Z, yaw, pitch, and roll directions). Hereinafter, the data acquired by the IMU 112 is also referred to as 6DoF data. However, the sensor is not limited to the IMU 112 and can be modified in various ways as long as the sensor can acquire the information necessary for performing Simultaneous Localization and Mapping (SLAM) (such as the pose (also referred to as the orientation) and movement of the measurement device 10 relative to the ground) and can acquire data of a different type from the data obtained by the sensor 111. In addition, the IMU 112 can be omitted when data from the IMU 112 is not required in a process such as SLAM.

[0203] The processing unit 113 includes, for example, various information processing devices (such as a central processing unit (CPU) and a microprocessor (MPU)), controls each unit in the measurement device 10, and performs predetermined processing on the data acquired by each sensor 111, IMU 112, etc. The processing performed in the processing unit 113 will be described in detail below.

[0204] The recording unit 114 includes, for example, a random access memory (RAM), a flash memory, etc., and holds various types of information, such as programs and parameters for operating the processing unit 113, information received by the communication unit 115, data acquired by the sensor 111 or IMU 112, and data processed by the processing unit 113.

[0205] For example, the communication unit 115 includes transceivers in various communication channels such as a local area network (LAN), Bluetooth (registered trademark), infrared communication, and mobile communication, and establishes a communication channel with the communication unit 123 in the relay device 20 to perform data transmission and reception. The communication unit 115 can directly transmit data to and receive data from the communication device 30 connected to the information processing device 40 via a wired or wireless channel. In this case, the relay device 20 can be omitted, and the communication unit 115 and the communication device 30 can establish a communication channel.

[0206] (Relay device 20)

[0207] The relay device 20 includes a processing unit 121, a recording unit 122, a communication unit 123, and a power supply 124. For example, the processing unit 121, the recording unit 122, and the communication unit 123 are interconnected to enable mutual data transmission and reception via an internal bus. It should be noted that the relay device 20 can be, for example, an information processing terminal that can be carried by the user U, such as a mobile phone, a smart phone, a tablet terminal, a smart watch, or smart glasses.

[0208] The power supply 124 supplies power to each unit in the relay device 20. The power supply 124 may include a rechargeable secondary battery or a primary battery such as a dry battery.

[0209] The processing unit 121 includes various information processing devices (such as a CPU and an MPU), controls each unit in the relay device 20, and relays the transmission and reception of data and various information between the information processing device 40 and the measurement device 10. In addition, the processing unit 121 can perform predetermined processing on the data and various types of information received from the measurement device 10 and / or the information processing device 40.

[0210] The recording unit 122 includes, for example, a RAM, a flash memory, etc., and holds programs and parameters for making the processing unit 121 function and various types of information received via the communication unit 123.

[0211] The communication unit 123 includes a transceiver capable of establishing a communication channel with the communication unit 115 of the measurement device 10 and the communication device 30, and sends various types of information to the communication unit 115 and the communication device 30 and receives various types of information from the communication unit 115 and the communication device 30. There may or may not be a difference between the communication channel established between the communication unit 123 of the measurement device 10 and the communication unit 115 and the communication channel established between the communication unit 123 and the communication device 30. For example, Bluetooth is allowed for data transmission / reception between the communication unit 123 and the communication unit 115 of the measurement device 10, and the use of LAN is allowed for data transmission / reception between the communication unit 123 and the communication device 30.

[0212] (Information processing device 40)

[0213] The information processing device 40 includes a processing unit 141, a recording unit 142, a communication unit 143, an image processing unit 144, and a power supply 145. For example, the processing unit 141, the recording unit 142, the communication unit 143, and the image processing unit 144 are connected to be able to perform mutual data transmission and reception via an internal bus. For example, the information processing device 40 may be an information processing device such as a personal computer (PC), a server, a smart phone, or a tablet terminal.

[0214] The power supply 145 supplies power to each unit in the measurement device 10. The power supply 116 may include a rechargeable secondary battery or a primary battery such as a dry battery.

[0215] The processing unit 141 includes various information processing devices such as a CPU and an MPU, for example, controls each unit in the information processing device 40, and performs predetermined processing on the data received from the measurement device 10 via the relay device 20. For example, the processing unit 141 estimates the three-dimensional positions of each of the measurement devices 10AL and 10AR (in this embodiment, the three-dimensional positions of the left and right wrists) based on the information acquired from each of the measurement devices 10AL and 10AR. Based on the estimated results, that is, the self-position estimation results of each of the measurement devices 10AL and 10AR and the data acquired from each of the measurement devices 10AL and 10AR (for example, the image data of the user U), the processing unit 141 measures the three-dimensional positions of other parts of the user U (for example, three-dimensional positions such as the left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, and head). Subsequently, based on the positions of each measured unit, the processing unit 141 controls the posture, movement, etc. of the avatar placed in the virtual space. Details of the processing executed in the processing unit 141 will be described below.

[0216] The recording unit 142 includes, for example, a RAM, a flash memory, etc., and holds various types of information such as programs and parameters for making the processing unit 141 function, information received via the communication device 30, and data processed by the processing unit 141.

[0217] The communication unit 143 includes transceivers in various communication channels such as a local area network (LAN), Bluetooth, infrared communication, and mobile communication, for example, and establishes a communication channel with the communication device 30 such as a wireless LAN adapter to perform data input and output. Note that the communication unit 143 may establish a direct communication channel with the relay device 20 or the measurement device 10. In this case, the communication device 30 may be omitted.

[0218] The image processing unit 144 includes an information processing device such as a graphics processing unit (GPU), for example, and performs image processing for generating an image to be displayed on the display device 50. For example, as Figure 3 shown, the image processing unit 144 performs rendering in the virtual space based on the processing results executed by the processing unit 141 to generate a video of the virtual humanoid C moving in the virtual space, and outputs the generated video to the display device 50 to display the video on the display device 50. The display device 50 may be incorporated in the same housing as the information processing device 40 (such as a notebook PC or a laptop PC), or may be separated from the information processing device 40. As long as there is no contradiction, the image and video can be appropriately paraphrased. Imaging and image capture can also be rewritten as appropriate.

[0219] 1.3 Schematic Configuration Embodiment of the Measurement Device

[0220] Next, configuration embodiments of the measurement device 10 according to the present embodiment will be described through some embodiments.

[0221] 1.3.1 First Embodiment

[0222] Figure 4 is an external view showing a schematic configuration embodiment of the measurement device according to the first embodiment. The first embodiment will describe an exemplary case including two sensors 111. In Figure 4 , (A) is a front view / rear view observed from a direction perpendicular to the extending direction of the wrist to which the measurement device 10 is attached; (B) is a top view / bottom view viewed from the extending direction of the wrist; (C) is a right side view / left side view when (A) is illustrated as a front view / rear view.

[0223] As Figure 4 shown, in addition to the configuration shown in Figure 2 , the measurement device 10 includes: a housing 101-1 that houses one of the two sensors 111; a housing 101-2 that houses the other sensor 111; and a band 103 that couples the housings 101-1 and 101-2 to each other and fixes the measurement device 10 to the wrist of the user U. In addition, the housings 101-1 and 101-2 are provided with lenses 102-1 and 102-2 (hereinafter, when the lenses are not distinguished, their reference numerals are described as "102") placed on the light receiving surfaces of the sensors 111.

[0224] The sensor 111 and the lens 102-1 provided in the housing 101-1 may be, for example, sensors that image the external direction of the wrist, and the sensor 111 and the lens 102-2 provided in the housing 101-2 may be, for example, sensors that image the internal direction of the wrist. In this way, the sensor 111 and the lens 102 provided in the housing 101-1 and the sensor 111 and the lens 102-2 provided in the housing 101-2 are configured to capture images on opposite sides, so that the entire part in the circumferential direction of the wrist can be set as the detection range.

[0225] To set the entire part in the circumferential direction of the wrist as the detection range by the sensor 111 and the lens 102 provided in the housing 101-1 and the sensor 111 and the lens 102-2 provided in the housing 101-2, as shown by the dashed line in (A) of Figure 4 , it is preferable to use a lens having a viewing angle (field of view (FOV)) of more than 180° and less than 270° as each lens 102.

[0226] In this way, by using the lens 102 with a FOV of 180° or greater, a full-body image of the user U can be captured regardless of the changes in the position and posture of the wrists of the user U each equipped with the measuring device 10AL or the measuring device 10AR.

[0227] It should be noted that the FOV of each lens 102 is not limited to the range above 180° and less than 270°, and can be above 90° and less than 180°, such as a general wide-angle lens or a fish-eye lens, or can be above 60° and less than 90°, such as the lens commonly used in a smart phone or a single-lens reflex camera. That is, various changes can be made as long as each measuring device 10 achieves a FOV of approximately 360° to be able to cover the entire surrounding environment of the wrist as the detection range.

[0228] 1.3.2 Second Embodiment

[0229] When a lens with a FOV less than 180° is used as the lens 102, it is permissible to provide multiple sets of sensors 111 and lenses 102 in each of the housing 101-1 facing the outside of the wrist and the housing 101-2 facing the inside of the wrist. Figure 5 is an external view showing a schematic configuration example of the measuring device according to the second embodiment, showing an embodiment of the measuring device in which multiple sets of sensors and lenses are provided in each housing. In the second embodiment, four sets of sensors 111 and lenses 102 are provided in each housing 101. In Figure 5 ,(A) is a front view / rear view observed from a direction perpendicular to the extending direction of the wrist to which the measuring device 10 is attached; (B) is a top view / bottom view seen from the extending direction of the wrist; (C) is a right side view / left side view when (A) is illustrated as a front view / rear view.

[0230] As Figure 5 shown, in the second embodiment, each housing 101 has a shape protruding in a quadrangular pyramid shape on the side opposite to the band 103, and one lens 102 is provided on each inclined surface of the quadrangular pyramid. For example, the optical axis of each lens 102 can be oriented in a direction perpendicular to the inclined surface.

[0231] In this way, as Figure 5 shown by the dashed line in (A) of , by arranging multiple (four in this example) lenses 102 such that the optical axes are inclined to each other, the entire surrounding directions of the wrist can be covered by the housing 101-1 and the housing 101-2 as the detection range. This enables full-body imaging of the user U to be performed regardless of any changes in the position and posture of the wrists of each user U equipped with the measuring device 10AL or the measuring device 10AR.

[0232] It should be noted that the FOV of each lens 102 in the second embodiment can be more than 90° and less than 180°, but is not limited thereto, and can be more than 180° and less than 270°, or can be more than 60° and less than 90°. That is, as long as a substantially 180° or larger FOV is achieved by combining multiple lenses 102 in each of the housings 101-1 and 101-2 so that each measuring device 10 can achieve a substantially 360° FOV to cover the entire surrounding environment of the wrist as the detection range, various changes can be made.

[0233] 1.3.3 Third Embodiment

[0234] Figure 6 is an external view showing a schematic configuration example of the measuring device according to the third embodiment, showing another example of the measuring device in which multiple sets of sensors and lenses are provided in each housing. In the third embodiment, five sets of sensors 111 and lenses 102 are provided in each housing 101. In Figure 6 , (A) is a front view / rear view observed from a direction perpendicular to the extending direction of the wrist to which the measuring device 10 is attached; (B) is a top view / bottom view seen from the extending direction of the wrist; (C) is a right side view / left side view when (A) is illustrated as a front view / rear view.

[0235] As Figure 6 shown, in the third embodiment, each housing 101 has a quadrangular prism shape such as a regular quadrangular prism, and one lens 102 is provided on each of the upper surface (the surface facing the side of the band 103) and the four side surfaces. For example, the optical axis of each lens 102 can be oriented in a direction perpendicular to each surface.

[0236] In this way, as Figure 6 shown by the dashed line in (A) of, by providing multiple (five in this example) lenses 102 such that the optical axes are inclined 90° to each other, the entire circumferential direction of the wrist can be covered by the housings 101-1 and 101-2 as the detection range. This makes it possible to image the entire body of the user U regardless of any changes in the position and posture of the wrist of the user U to which each of the measuring devices 10AL and 10AR is attached.

[0237] Note that the FOV of each lens 102 in the third example can be 90° or more and less than 180°, but is not limited thereto, and can be 180° or more and less than 270°, or can be 60° or more and less than 90°. That is, various changes can be made as long as a FOV of approximately 180° or more is achieved by combining multiple lenses 102 in each of the housings 101-1 and 101-2 so that each measuring device 10 can achieve a FOV of approximately 360° to cover the entire surrounding environment of the wrist as the detection range.

[0238] 1.4 Position measurement method for each part

[0239] Next, a method for measuring the position of each part of the user U according to the present embodiment will be described in detail with reference to the accompanying drawings. In the present embodiment, the position measurement parts are represented by the left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, and head, but the position measurement parts are not limited thereto, and various parts of the user U can be defined as the position measurement targets. In addition, for simplicity, left / right distinction will not be made inappropriately in the following description.

[0240] 1.4.1 Wrist

[0241] First, a method for measuring the positions of the two wrists of the user U will be described. In the present embodiment, the measuring devices 10AL and 10AR are attached to each wrist of the user U. The wrist positions of the user U are measured from the positions estimated by the measuring devices 10AL and 10AR themselves (self-position estimation results).

[0242] Figure 7 is a diagram illustrating a method for measuring the wrist position according to the present embodiment. Figure 7 The user U shown in wears the measuring device 10AL on their left wrist and the measuring device 10AR on their right wrist. In the figure, the dashed circles shown around each of the measuring devices 10AL and 10AR represent the detection ranges in which each measuring device 10 can detect an object, as described above with reference to Figures 4 to 6 as described. As Figure 7As shown, each of the measurement devices 10AL and 10AR can image a wide range (preferably, around 360 degrees) around the position to which the measurement device is attached. In other words, objects located around the position where the user wears the sensor device basically over the entire surrounding environment at 360 degrees can be detected. Further, each of the measurement devices 10AL and 10AR includes an IMU 112 capable of detecting movement in the 6DoF directions (X, Y, Z, Yaw, Pitch, Roll directions), and thus includes a configuration necessary for performing SLAM. In the measurement device 10AL, an image sensor (an example of the sensor 111) of the measurement device 10AL acquires an image of, for example, the space in which the measurement device 10AL is provided, and this space is the space that forms the surrounding environment of each device. Alternatively, as described below, a distance measurement sensor (an example of the sensor 111) of the measurement device 10AL can acquire, for example, distance information, information on the distance from the measurement device 10AL to an object existing around the measurement device 10AL in the space including the measurement device 10AL. The same applies to the measurement device 10AR. In the present embodiment, each of the measurement device 10AL and the measurement device 10AR uses the SLAM technique to estimate the position and posture of the wrist to which the measurement device itself is attached.

[0243] Specifically, for example, SLAM is performed using the image data acquired by one or more sensors 111 included in the measurement device 10AL attached to the left wrist and using the 6DoF data acquired by the IMU 112, whereby the self-position of the measurement device 10AL (i.e., the left wrist) in the three-dimensional coordinate system is estimated. Similarly, SLAM is performed using the image data acquired by one or more sensors 111 included in the measurement device 10AR attached to the right wrist and using the 6DoF data acquired by the IMU 112, whereby the self-position of the measurement device 10AR (i.e., the right wrist) in the three-dimensional coordinate system is estimated.

[0244] Except for the coordinate system integration process described below, the objects imaged by each sensor 111 of the measurement device 10AL and the measurement device 10AR are not limited. That is, the user U can image any subject having the measurement devices 10AL and 10AR attached to both wrists while performing SLAM.

[0245] The coordinate systems of SLAM performed using the left wrist measurement device 10AL and the coordinate systems of SLAM performed using the right wrist measurement device 10AR are integrated into a common coordinate system. For example, the coordinate system integration process can be executed by the information processing device 40. By integrating the coordinate systems of the measurement device 10AL and the measurement device 10AR attached to the two wrists, the estimated own position of the measurement device 10AL (i.e., the position of the left wrist) and the estimated own position of the measurement device 10AR (i.e., the position of the right wrist) can be regarded as positions on the common coordinate system. The position estimation of the measurement devices 10AL and 10AR enables the measurement of the positions of the left and right wrists of the user U.

[0246] 1.4.2 Elbows and Shoulders

[0247] Next, a method for measuring the positions of the two elbows and two shoulders of the user U will be described. Figure 8 FIG. is a diagram showing a method for measuring the positions of the elbows and shoulders according to the present embodiment. Note that the shape of the skeleton model of the human body and the positions of the elbows and shoulders located between the wrists and a reference part (e.g., the torso) in the user U can be measured based on the skeleton model of the human body, as described below.

[0248] (Elbow)

[0249] As described above, SLAM using the measurement device 10 attached to the wrist J1 knows the own position of the wrist J1 of the user U. Therefore, the own position of the ulna including the wrist J1 of the user U can also be regarded as known.

[0250] On the other hand, for example, the length L1 of the ulna of the user U is known from a general human body skeleton model or a user-specific skeleton model constructed by measuring the user U. Therefore, as Figure 8 shown, the position of the elbow J2 of the user U can be measured by moving the position of the measurement device 10 a distance L1 in the ulna extension direction based on the posture of the measurement device 10.

[0251] (Shoulder)

[0252] In addition, the length L2 of the humerus of the user U is also known from the skeleton model. The shoulder J3 of the user U exists on a spherical surface Q1 centered on the estimated position of the elbow J2 and has a radius equal to the humerus length L2.

[0253] On the other hand, when the self-position of a reference point (e.g., a bib attached to the suitcase, etc.) in the suitcase B of the user U (hereinafter referred to as the starting point O) is known, and the distance L3 from the starting point O to the shoulder J3 is known, the shoulder J3 of the user U exists on the circle Q2 obtained as the intersection point, which is created when a spherical surface centered at the origin 0 with a radius of the distance L3 intersects a plane passing through the origin 0 and satisfying Z = 0 (hereinafter, also referred to as the Z0 plane).

[0254] Therefore, the position of the shoulder J3 of the user U is narrowed down to two points where the spherical surface Q1 intersects the circle Q2.

[0255] Here, the existence range of the shoulder J3 relative to the origin O can be restricted from the skeleton model to Y > 0. By this operation, the position of the shoulder J3 of the user U can be further narrowed down to a point that satisfies the condition Y > 0 among the above two narrowed-down points.

[0256] 1.4.3 Knees and Ankles

[0257] Next, a method for measuring the positions of both the knees and ankles of the user U will be described. Fig. 9 FIG. is a diagram showing a method for measuring the positions of the knees and ankles according to the present embodiment. Fig.10 FIG. is a diagram for explaining position measurement using triangulation.

[0258] As described above, SLAM knows the self-position of the wrist J1 of the user U using the measuring device 10 attached to the wrist J1. In addition, the posture (orientation in the yaw, pitch, and roll directions) of the measuring device 10 is understood from the measurement results of the IMU 112 mounted on the measuring device 10.

[0259] In addition, as Fig. 9 shown, the measuring device 10AL attached to the left wrist of the user U and the measuring device 10AR attached to the right wrist acquire image data including images of the right knee J4 and left knee J4, right ankle J5 and left ankle J5 of the user U.

[0260] Therefore, the directions in which the left ankle J5 and right ankle J5 of the user U exist relative to the measuring device 10AL in each of the left knee J4 and right knee J4 can be specified from the image data acquired by the measuring device 10AL. Similarly, the directions in which the left ankle J5 and right ankle J5 of the user U exist relative to the measuring device 10AR in each of the left knee J4 and right knee J4 can be specified from the image data acquired by the measuring device 10AR.

[0261] In this way, the self-positions of the two measuring devices 10AL and 10AR are known, and the directions in which the left knee J4 and right knee J4, left ankle J5 and right ankle J5 of the user U exist relative to each measuring device 10 can be specified. Therefore, as Fig.10 As shown, the positions of the left knee J4 and right knee J4, and left ankle J5 and right ankle J5 of user U can be measured using triangulation.

[0262] In the case of a frame where at least one of the left knee J4 and right knee J4, and left ankle J5 and right ankle J5 of user U cannot be captured by the sensor 111 of at least one of the measuring devices 10AL and 10AR, it is allowed to interpolate the positions of the missing parts in the image data of this frame by using techniques such as frame interpolation.

[0263] 1.4.4 Head and torso

[0264] Next, a method for measuring the positions of the head and torso of user U will be described. Fig.11 It is a diagram for showing a method of measuring the positions of the head and trunk according to the present embodiment.

[0265] Similar to the above knee and ankle position measurement method, the self-position of the wrist J1 of user U is obtained by SLAM using the measuring device 10, and the posture of the measuring device 10 (orientations in the yaw, pitch, and roll directions) is obtained from the measurement results of the IMU 112. In addition, as Fig.11 shown, the measuring devices 10AL and 10AR attached to the two wrists of user U each acquire image data including images of the feature points of the head H (forehead, eyes, nose, mouth, chin, etc.) and the feature points of the torso B of user U (for example, bib, pocket, button, collar, etc.).

[0266] This makes it possible to specify in which directions the feature points of the head H (forehead, eyes, nose, mouth, chin, etc.) and the feature points of the torso B of user U (for example, bib, pocket, button, collar, etc.) exist with respect to each measuring device 10 from the image data separately acquired by the measuring devices 10AL and 10AR, enabling the measurement of the individual positions using triangulation as shown in Fig.10 .

[0267] Similar to the knee and ankle, in the case of a frame where at least one of the feature points of the head H and torso B of user U cannot be captured by the sensor 111 of at least one of the measuring device 10AL and the measuring device 10AR, it is also allowed to interpolate the positions of the missing feature points in the image data of this frame by using techniques such as frame interpolation.

[0268] 1.4.5 Another method for measuring the position of each site

[0269] Although the above measurement method is an exemplary case of measuring the position of each part of the user U by triangulation from the image data imaged by two measurement devices 10 with known self-positions, the measurement method according to the present embodiment is not limited thereto, and various modifications can be made. For example, when at least one sensor 111 in at least one measurement device 10 with known self-position is a distance measurement sensor, the position of each part of the user U relative to the measurement device 10 with known self-position can be measured by using the depth data acquired by the sensor 111 and the IMU 112.

[0270] Fig.12 is a view showing the position measurement using a distance measurement sensor. In the case of the present embodiment, the self-positions of the measurement device 10AL attached to the left wrist and the measurement device 10AR attached to the right wrist are known individually by their SLAM. Therefore, by using at least one of the sensors 111 included in the measurement device 10AL and the measurement device 10AR as a distance measurement sensor, the depth data acquired by using the sensor 111 can be obtained. When the measurement device 10AL is described as an example, the measurement of the distance between each part of the user U (for example, a certain part of the user U, such as the left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, or head) and the measurement device 10AL capturing the part has been performed. In addition, the orientation (orientation in the yaw, pitch, and roll directions) of the distance measurement sensor of the measurement device 10AL is known from the measurement result of the IMU 112. Therefore, the direction and distance between each part of the user U and the measurement device 10AL can be grasped. Since the self-position of the measurement device 10AL is also known, the position (three-dimensional coordinates) of each part of the user U is known. The same applies to the measurement device 10AR. In the above Fig.10 example, the positions of the left knee J4 and the right knee J4, and the left ankle J5 and the right ankle J5 of the user U can be estimated using only one of the measurement device 10AL and the measurement device 10AR, or can be estimated using both devices.

[0271] 1.4.6 Method for Measuring the Shape of the Hand / Fingers

[0272] Figures 13 to 16 is a view showing the method for measuring the shape of the hand / fingers according to the present embodiment. As Figures 13 to 16 shown, when at least one sensor 111 of the measurement device 10 attached to the wrist can image the hand of the user U, the posture of the palm or each finger can be measured from the shape and / or distance information of the hand / fingers captured as an image. For example, information such as whether the palm is open or curled, which finger is extended, or which finger is curled / bent can be measured from the image data (which can be depth data) acquired by the sensor 111.

[0273] The sensor 111 for imaging the hand of user U can be implemented by using sensors such as image sensors (including IR sensors and light sources), range sensors, and EVS. In addition, the sensor for imaging the hand is not limited to the sensor 111 mounted on the measuring device 110, and can be a dedicated sensor separately provided for imaging the hand.

[0274] For example, the sensor 111 or the dedicated sensor can capture an image of the hand from the palm side, as Fig.13 shown, can capture an image of the hand from the back side of the hand, as Fig.14 shown, can capture an image of the hand from the thumb side, as Fig.15 shown, or can capture an image of the hand from the little finger side, as Fig.16 shown. Alternatively, Figures 13 to 16 two or more of the imaging techniques shown in

[0275] In addition, the shape of the hand / finger of user U can be measured from the image data obtained by imaging the hand of user U by using a sensor attached to a part other than the wrist of user U (e.g., head, torso, etc.) or a sensor mounted on a part other than user U.

[0276] In addition, the shape of the hand of user U is not limited to the above method using an optical sensor (such as the sensor 111 or the dedicated sensor), and can also be measured by using a method using a non-optical sensor (such as an electromyographic sensor, a bending sensor, or an IMU). For example, as Fig.17 shown, non-optical sensors 111g such as electromyographic sensors, bending sensors, or IMUs can be provided at finger joint parts, muscle parts, fingertip parts, etc. in a glove-like measuring device 10G, and the shape of the hand of user U can be measured based on the data obtained by the non-optical sensors 111g.

[0277] It should be noted that similar to the measurement process of the position of each unit of user U, the measurement process of the hand shape can be basically executed in the information processing device 40. However, the execution of the process is not limited to this, and can be executed in the measuring device 10, the relay device 20, etc.

[0278] 1.5 Coordinate system

[0279] Next, an outline of the process of integrating the coordinate systems of the measuring device 10AL and the measuring device 10AR into a common coordinate system will be described. Fig.18 is a flowchart showing a schematic operation example of the coordinate system integration process according to the present embodiment. The coordinate integration process illustrated below is only an example, and can be modified in various ways.

[0280] As Fig.18 shown, in this operation, first, the power supplies of each measurement device 10 (in this example, two wrists) connected to a predetermined part of user U, that is, the power supplies of measurement devices 10AL and 10AR are turned on (step S1), and each measurement device 10 starts imaging through sensor 111 (step S2). At that time, each measurement device 10 can also start collecting 6DoF data by IMU 112.

[0281] Next, each measurement device 10 starts SLAM using the image data collected by sensor 111 and the 6DoF data collected by IMU 112 (step S3). Note that at this stage, the coordinate system of the three-dimensional map used in SLAM (hereinafter also referred to as the initial coordinate system) can be based on the position and orientation of each measurement device 10 at the start of SLAM as the starting point.

[0282] Next, each measurement device 10 starts the process of transmitting its own position (coordinates and orientation) managed by SLAM to information processing device 40 (step S4). Note that the own position information sent from each measurement device 10 can be sent to information processing device 40 via relay device 20.

[0283] Then, each measurement device 10 starts the recognition process of the image data acquired by sensor 111 (step S5).

[0284] Next, as a result of the recognition process in step S5, each measurement device 10 determines whether sensor 111 has imaged a specific marker (step S6). The markers recognized by measurement device 10 can be the same marker.

[0285] Here, the specific marker can be, for example, a marker that can specify the orientation relative to each measurement device 10 in the captured image by including three or more feature points (such as a graphic, a QR code, or a three-dimensional object). Specifically, the marker can be various markers that can be guiding markers, such as a graphic or a QR code installed in the real space, a graphic or a QR code displayed on the screen of a fixed display device, a graphic or a QR code projected on a specific surface such as a wall, or an object fixed at a specific position.

[0286] Next, when a specific marker is imaged as a new origin (hereinafter, also referred to as the marker origin), each measurement device 10 in the measurement device sets its own position managed by SLAM (step S7), and sets a coordinate system with the marker origin as the origin (hereinafter, also referred to as the client map coordinate system or the first coordinate system) on the map data used in SLAM (step S8). As a result, the self-position sent from each measurement device 10 starting from step S4 becomes the self-position in the client map coordinate system. The start of SLAM in each measurement device 10 is not limited to step S3 described above, and can be set after setting the client map coordinate system in step S8.

[0287] Next, each measurement device 10 specifies the coordinate information of at least three feature points in the marker identified in step S5 in the client map coordinate system, and sends the specified coordinate information to the information processing device 40 via the relay device 20 (step S9).

[0288] In response to this operation, based on the coordinate information of the marker received from each measurement device 10, the information processing device 40 generates a transformation matrix for integrating the client map coordinate system used in SLAM in each measurement device 10 into a common coordinate system (hereinafter, also referred to as the server map coordinate system or the second coordinate system) in each measurement device 10 (step S10). For example, the information processing device 40 generates a transformation matrix for achieving the matching between the coordinates of each feature point of the marker received from one of the measurement devices 10AL and 10AR and the coordinates of each feature point of the marker received from the other measurement device.

[0289] Thereafter, the information processing device 40 uses the generated transformation matrix to change the self-position of each measurement device 10 sent from each measurement device 10 to the position on the server map coordinate system. By this operation, the coordinate systems of the respective measurement devices 10 are integrated into the common server map coordinate system (step S11). When the coordinate systems of the respective measurement devices 10 are integrated into the common server map coordinate system in this way, this operation ends.

[0290] Although this specification has described an exemplary case where SLAM is executed in each of the measurement devices 10 and the coordinate integration process is executed in the information processing device 40, the operation is not limited thereto. Each measurement device 10's SLAM and coordinate integration process can be executed in any one of the measurement device 10, the relay device 20, and the information processing device 40.

[0291] As described above, by integrating the coordinate system of the map data used in each measurement device 10 into the common server map coordinate system, for example, in the case of an application that operates an avatar in a virtual space, the user U can operate the avatar at the point where the integration process of the coordinate systems is successful. By confirming the enabling operation of the avatar via the display device 50, the user U can recognize that the coordinate system integration process has been successful.

[0292] In the case where the user U needs to know whether the coordinate system integration is successful at the start of a physical activity (such as an application for observing the physical activity posture), it is allowed to equip the housing of each measurement device 10 with a display device, such as a light-emitting diode (LED) that notifies the user of the completion / incompletion of the coordinate system integration process. Alternatively, each measurement device 10 can notify the user through an announcement such as "Please execute the process" or through audio "The process is completed".

[0293] 1.6 Schematic operation embodiment

[0294] Next, a schematic operation embodiment of the human body tracking system 1 according to the present embodiment will be described. Fig.19 It is a diagram showing an embodiment in which a user wearing a measurement device exists in the space in this specification. Fig. 20 It shows Fig.19 an embodiment of the set of coordinate systems of the space shown in

[0295] As Fig.19 and Fig. 20 shown, in this specification, the three-dimensional map can be a three-dimensional environmental map (also referred to as map data) generated from image data and generated by SLAM executed in each measurement device 10 (i.e., separately in the measurement device 10AL and the measurement device 10AR).

[0296] As Fig. 20 shown, the client map coordinate system (including the above-mentioned coordinate systems) can be the coordinate systems CR and CL of the three-dimensional map, where each measurement device 10 independently determines its origin and XYZ directions. Therefore, even when the measurement device 10 moves, the origin of the client map coordinate system can be fixed to a point on the map space.

[0297] As Fig. 20 shown, the server map coordinate system can be the coordinate system CC of the three-dimensional map for integration, where the information processing device 40 determines the origin and XYZ directions, and can be the coordinate system finally used in applications such as for controlling avatars, etc.

[0298] Furthermore, the client map coordinate system can be a coordinate system having the origin at the position of each measurement device 10. Therefore, the origin of the client map coordinate system can change as the measurement device 10 moves.

[0299] 1.6.1 Flowchart

[0300] Fig.21 is a flowchart showing a schematic operation example of the human body tracking system according to the present embodiment.

[0301] As Fig.21 shown, in this operation, the first step is to obtain a three-dimensional map (also referred to as an initial three-dimensional map) for SLAM of each measurement device 10 in the information processing device 40 or each measurement device 10 (i.e., individually in the measurement devices 10AL and 10AR) (step S101). Embodiments of the method for obtaining the initial three-dimensional map may include various methods, such as a method of turning an existing three-dimensional map and a method of integrating three-dimensional maps created in SLAM independently executed by each measurement device 10.

[0302] Next, SLAM is performed for each measurement device 10 (step S102). In the SLAM of each measurement device 10, the initial three-dimensional map created in step S101 can be updated for each measurement device 10.

[0303] Next, based on the information acquired by the sensors 111 and IMUs 112 of each measurement device 10, the positions (coordinate information) of each part (left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, head, etc.) of the user U are measured in the server map coordinate system (step S103). Additionally, in step S103, the orientation (posture) of each part can also be measured.

[0304] For example, the position (coordinate information) of each part measured in this way is input into an application that links an avatar, a person, etc. to the body movements of the user U (step S104). This enables the operation of a human body model such as an avatar or a person displayed on the display device 50 according to the actions of the user U.

[0305] Thereafter, for example, in the information processing device 40, it is determined whether to end the current operation (step S105). When it is determined that the current operation ends (in step S105, "yes"), the current operation ends. On the contrary, when the current operation does not end (in step S105, "no"), the current operation returns to step S102 to perform subsequent operations.

[0306] 1.6.2 Variations of the Initial Three-Dimensional Map Acquisition Method

[0307] Here, some variations of the method for obtaining the initial three-dimensional map shown in step S101 Fig.21 are illustrated.

[0308] (Using an existing three-dimensional map)

[0309] The method of transferring an existing three-dimensional map can be implemented by adopting various methods, such as a method of transferring existing map data as an initial three-dimensional map and a method of transferring a three-dimensional position map created by previously executed SLAM or the like as an initial three-dimensional map. In this case, the system in which SLAM has been previously executed is not limited to the human body tracking system 1 and can be a different system.

[0310] (Integration of three-dimensional maps generated by each measurement device)

[0311] In the method of independently executing by each measurement device in the measurement device 10 for integrating the three-dimensional map created in SLAM, for example, individual three-dimensional maps can be integrated to create an initial three-dimensional map at a stage where each measurement device in the measurement device 10 has created its three-dimensional map to a certain extent.

[0312] Fig. 22 is a flowchart showing an example of the process of map integration processing according to the present embodiment. Figure 23 to Figure 25 is for supplementing Fig. 22 shown in the process. In Fig. 22 For the sake of generality, the number of measurement devices 10 is set to n.

[0313] For example, when the user U wearing the measurement device 10AL and the measurement device 10AR is present in Fig.19 the space SP shown in Fig. 22 As shown in steps S111a to S111n of

[0314] each measurement device 10 (that is, the measurement device 10AL and the measurement device 10AR) independently executes SLAM to create an initial three-dimensional map based on the image data acquired by the sensor 111 and the 6DoF data acquired by the IMU 112. Fig.23 and Fig.24 shown, SLAM is independently and separately executed in the measurement device 10AL and the measurement device 10AR to create a three-dimensional map.

[0315] Note that the coordinate system of the three-dimensional map created in the measurement device 10AL (refer to Fig.23 ) can be the client map coordinate system CL independently set in the measurement device 10AL. The coordinate system of the three-dimensional map created in the measurement device 10AR (see Fig.24 ) can be the client map coordinate system CR independently set in the measurement device 10AR.

[0316] Next, each measurement device 10 determines the extent to which a three-dimensional map can be created at the current point such that it can be integrated with the three-dimensional map created by another measurement device 10 (steps S112a to S112n). For example, each measurement device 10 can determine whether integration is possible based on criteria such as whether a predetermined marker exists in the created three-dimensional map, whether there is one or more objects to be landmarks, whether there is one or more characteristic shapes, or whether there are two or more walls, ceilings, etc. that are not parallel to each other and can be specified.

[0317] When the determination results in steps S112a to S112n indicate that three-dimensional map integration processing can be performed for all measurement devices 10 ( "Yes" in steps S112a to S112n), the process of integrating the three-dimensional maps created by each measurement device 10 is performed (steps S113a to S113n), and the process proceeds to Fig.21 step S102 shown in

[0318] In the case where the measurement devices 10AL and 10AR as in the present embodiment are attached to each wrist of the user U, as Fig.25 shown, the three-dimensional map created by the measurement device 10AL (reference Fig.23 ) and the three-dimensional map created by the measurement device 10AR (reference Fig.24 ) are integrated to create a common three-dimensional map (initial three-dimensional map).

[0319] In steps S113a to S113n, before the measurement device 10 is fully prepared for map integration, the map integration process can be sequentially performed on the measurement devices 10 that have achieved the ability to integrate three-dimensional maps, and the process can proceed to Fig.21 step S102 in

[0320] 1.6.3 Specific Techniques for Map Integration

[0321] Next, specific techniques for map integration performed in steps S113a to S113n in Fig. 22 will be described through some embodiments.

[0322] (First Technique)

[0323] The first technique will be described as an exemplary case where the three-dimensional map of the measurement device 10 is integrated with a predetermined marker fixed in the external environment (e.g., in the space SP) as a reference. That is, in the first technique, when a predetermined marker is recognized in steps S112a to S112n in Fig. 22 , it is determined that integration is possible. Fig.26 is a diagram for explaining map integration according to the first technique.

[0324] In Fig.26 In the illustrated embodiment, user U uses the left wrist measurement device 10AL and the right wrist measurement device 10AR to image the marker MK fixed to the floor in the space SP, for example, separately. Here, the three-dimensional position (coordinates) and pose (orientation) of each measurement device 10 in each client map coordinate system are known from the SLAM performed by each measurement device 10. In addition, based on the image data acquired by the sensor 111 of each measurement device 10, it is also possible to obtain in which direction the marker MK is positioned relative to each measurement device 10.

[0325] Thus, by specifying the positions of the respective measurement devices 10 relative to the similar marker MK existing in the real space (space SP), the correspondence between the client map coordinate systems of the multiple measurement devices 10 can be obtained. This enables the integration of the three-dimensional maps created by the individual measurement devices 10 into a common initial three-dimensional map.

[0326] Although Fig.26 an exemplary case where two measurement devices 10 are attached to the two wrists of user U is shown, the positions are not limited to this example. For example, as Fig. 27 shown, even when at least one of the multiple measurement devices 10 (measurement device 10FL) is attached to a position different from the wrist of user U (in this example, the left ankle), the correspondence between the client map coordinate systems of the multiple measurement devices 10 can be obtained by a similar technique, enabling the integration of the three-dimensional maps created by the individual measurement devices 10 into a common initial three-dimensional map.

[0327] (Variation of the first technique)

[0328] In the above first technique, an example of the case where the correspondence between the client map coordinate systems of the multiple measurement devices 10 is obtained with reference to the specified marker MK fixed to the external environment (such as space SP) is described, but the object used as a reference is not limited to the marker MK, and various variations can also be made. For example, as described above, one or more objects serving as landmarks, one or more characteristic shapes, two or more walls or ceilings that are not parallel to each other and can be specified, etc. can also be used as references.

[0329] Fig.28 is a flowchart of an embodiment showing the process of map integration processing according to a variation of the first technique. For example, the operations shown in Fig.28 can be executed instead of the operations shown in Fig. 22 .

[0330] As Fig.28 shown, in this technique, first, as in Fig. 22In steps S111a to S111n, SLAM for creating an initial three-dimensional map is performed individually in each measurement device 10 (step S121).

[0331] Next, the information processing device 40 acquires image data obtained by at least one sensor 111 of each measurement device 10 via the relay device 20 (step S122).

[0332] Next, the information processing device 40 performs recognition processing on the image data acquired from each measurement device 10 (step S123).

[0333] As a result of the recognition processing in step S123, it is determined whether the same object is captured in the image data acquired from two or more or all of the measurement devices 10 (step S124). When the same object is captured (Yes in step S124), this operation proceeds to step S125. On the other hand, when the same object is not captured (No in step S124), this operation returns to step S122, and the subsequent operations are repeated.

[0334] In step S125, the information processing device 40 performs, for example, semantic segmentation on the respective image data acquired from each measurement device 10. Subsequently, in step S125, the information processing device 40 extracts feature points of each image data labeled and classified for each pixel through semantic segmentation (step S126), and specifies the correspondence between the image data of the extracted feature points to determine corresponding pixels across individual image data (step S127).

[0335] Various techniques can be adopted to determine corresponding pixels between individual image data. Examples of such techniques include the following technique: As a result of semantic segmentation in step S125, corresponding feature points on each image data are determined by performing polygon approximation on regions associated with similar labels or categories in each image data, and the pixels located at each corresponding feature point are determined as corresponding pixels.

[0336] In this way, similar to the above first technique, by determining corresponding pixels between image data, the correspondence between the client map coordinate systems of the multiple measurement devices 10 can be obtained. Therefore, based on the obtained correspondence between the client map coordinate systems, the information processing device 40 performs map integration processing for integrating the three-dimensional maps created by the individual measurement devices 10 into a common initial three-dimensional map (step S128), and proceeds to Fig.21 step S102 shown in

[0337] (Second technique)

[0338] As a second technique, an example is described in which, instead of the marker MK fixed to the external environment according to the first technique, three or more markers provided on the outer surfaces of a plurality of measurement devices 10 are used to directly obtain the positional relationship of the measurement devices 10, that is, the correspondence between the client map coordinate systems of the plurality of measurement devices 10. Fig.29 And Fig.30 is a diagram for showing map integration according to the second technique.

[0339] In Fig.29 In the example shown, three markers M1 to M3 are provided on a surface visually recognizable from one direction in the housing of the measurement device 10AR attached to one wrist (in this example, the right wrist). The markers M1 to M3 on the surface of the measurement device 10AR are imaged from the measurement device 10AL attached to the other wrist (in this example, the left wrist). By this operation, as Fig.30 shown, the relationship between the markers m1 to m3 on the image data IM1 and the markers M1 to M3 in the actual space can be represented by a total of six expressions, two expressions for each of the markers M1 to M3, as in the following formula (1). In the following formula, "i" is a variable corresponding to the markers M1 to M3. Therefore, in this embodiment, "i" is an integer from 1 to 3.

[0340]

[0341] Here, fx and fy represent the focal lengths in the horizontal and vertical directions of the image, and cx and cy represent the positions of the optical centers on the image. These values are device-specific and constant known values.

[0342] Here, since the positional relationship between the markers M1 to M3 provided in the measurement device 10AR is known, a total of three expressions represented by the following formula (2) can be established.

[0343]

[0344] Based on the total nine formulas represented by the above formulas (1) and (2), nine unknowns can be obtained, which are the coordinates of three points in the real space (the coordinates of the markers M1 to M3). By obtaining the coordinates of the three points (the coordinates of the markers M1 to M3), the relative positional relationship (including the posture) between the measurement devices 10 can be derived. This makes it possible to obtain the correspondence between the client map coordinate systems of the plurality of measurement devices 10, and as a result, the three-dimensional maps created by the individual measurement devices 10 can be integrated into a common initial three-dimensional map based on the correspondence between the client map coordinate systems.

[0345] The second technique is not limited to the case where two measurement devices 10AL and 10AR are attached to two wrists of the user U, and is equally applicable even when at least one of a plurality of measurement devices 10 (measurement device 10FL) is attached to a position different from the wrist of the user U (for example, the ankle).

[0346] (First modification example of the second technique)

[0347] The first modification example of the second technique will be described as an exemplary case where two markers M1 and M2 are provided in the measurement device 10 instead of three markers M1 to M3. Fig.31 and Fig.32 is a diagram for showing map integration according to the first modification example of the second technique.

[0348] In Fig.31 the illustrated embodiment, two markers M1 and M2 are provided on a surface visually recognizable from one direction in the housing of the measurement device 10AL attached to one wrist (in this embodiment, the left wrist). The markers M1 and M2 on the surface of the measurement device 10AL are imaged from the measurement device 10AR attached to the other wrist (in this example, the right wrist). By this operation, as Fig.32 shown in, the relationship between the markers m1 - m2 on the image data IM2 and the markers M1 - M2 in the actual space can be expressed by a total of four expressions, and for each of the markers M1 and M2, two expressions are from the above formula (1). In addition, since the positional relationship between the markers M1 and M2 provided in the measurement device 10AL is known, one expression can be established according to the above formula (2).

[0349] Furthermore, in the first modification example, since the angle θ between the straight line connecting the markers M1 and M2 and the direction of gravity (the direction of gravity detected by the IMU 112 is known), the following formula (3) can be established.

[0350]

[0351] Based on the total six formulas represented by the above formulas (1) to (3), six unknowns, which are the coordinates of two points (the coordinates of the markers M1 and M2) on the actual space, can be obtained. By obtaining the coordinates of two points (the coordinates of the markers M1 and M2) and the direction of gravity, the relative positional relationship (including the posture) between the measurement devices 10 can be derived. This makes it possible to obtain the correspondence between the client map coordinate systems of the plurality of measurement devices 10, and as a result, the three - dimensional maps created by the individual measurement devices 10 can be integrated into a common initial three - dimensional map based on the correspondence between the client map coordinate systems.

[0352] (Second modification example of the second technique)

[0353] In the above-described second technique and its modification example 1, by separately obtaining the distances between the respective measurement devices 10, it is possible to more accurately synthesize multi-dimensional maps.

[0354] The distance between two measurement devices 10 can be detected, for example, by using at least one of the sensors 111 included in at least one measurement device 10 as a distance measurement sensor. However, the detection of the distance is not limited thereto. For example, as Fig.33 shown, the intensity of the radio wave signal output from one measurement device 10 can be detected by another measurement device 10, so as to detect the distance based on the detected radio field intensity.

[0355] (Third Technique)

[0356] For example, a third technique will be described as a case where an initial three-dimensional map is created by integrating the point clouds of the three-dimensional maps created by each measurement device 10 by using a technique such as the iterative closest point (ICP) algorithm. Fig.34 is a diagram for explaining map integration according to the third technique. In Fig.34 it, the point cloud PCL represents the point cloud of the three-dimensional map created by the measurement device 10AL, and the point cloud RCR represents the point cloud of the three-dimensional map created by the measurement device 10AR.

[0357] In the third technique, first, as Fig.34 (A) of shows, each point of the point cloud PCL of the three-dimensional map created by the measurement device 10AL and each point of the point cloud PCR of the three-dimensional map created by the measurement device 10AR are associated with each other according to a specific reference. The specific reference can be determined by adopting various references such as the nearest neighbor points, color information assigned to each point as attribute information, and other feature quantities.

[0358] When the points between the point clouds are associated with each other in this way, the next step is to obtain a transformation matrix that minimizes the objective function (for example, the distance between corresponding points) between the associated points.

[0359] Next, as Fig.34 (B) of shows, by applying the transformation matrix to one point cloud (in this embodiment, the point cloud PCL), this point cloud is made closer to the other point cloud.

[0360] After that, this series of processes is repeated until the objective function converges, whereby the three-dimensional maps created by each measurement device 10 are integrated into a common initial three-dimensional map.

[0361] (Modification of the Third Technique)

[0362] In a modification of the third technique, two or more regions in the three-dimensional map created by each measurement device 10 are approximated as two or more non-parallel planes, and the client map coordinate system of the three-dimensional map is transformed such that the two or more created planes overlap each other across multiple three-dimensional maps. With this technique, the three-dimensional maps created by individual measurement devices 10 can also be integrated into a common initial three-dimensional map.

[0363] 1.6.4 Variations of the SLAM execution mode

[0364] Next, some variations will be illustrated for Fig.21 the execution form of SLAM for each measurement device 10 shown in step S102 of

[0365] For example, the SLAM of each measurement device 10 can be executed in each measurement device 10 or can be executed in the information processing device 40. Therefore, this description will describe exemplary cases, specifically, the case where the SLAM of each measurement device 10 is executed in the information processing device 40 (the first embodiment), the case where SLAM is executed using an initial three-dimensional map shared in each measurement device 10 (the second embodiment), and the case where SLAM is executed using a separate three-dimensional map in each measurement device 10 (the third embodiment).

[0366] (The first embodiment)

[0367] Fig.35 is a flowchart for explaining the process of SLAM processing according to the first embodiment. As Fig.35 shown, in the first embodiment, when obtaining the initial three-dimensional map in Fig.21 step S101 of

[0368] step S131, first, the information processing device 40 obtains the image data acquired by the sensor 111 of each measurement device 10 or the feature points obtained by performing recognition processing on the image data, and the 6DoF data acquired by the IMU 112 from each measurement device 10.

[0369] Next, based on the image data or feature points and the 6DoF data acquired from each measurement device 10, the information processing device 40 estimates the position of each measurement device 10 on the three-dimensional map in the server map coordinate system (step S132).

[0369] Next, based on the image data or feature points and the 6DoF data acquired from each measurement device 10, the information processing device 40 updates the three-dimensional map (step S133). After that, this operation proceeds to Fig.21 step S103 of

[0370] In this way, according to the method of performing SLAM for each measurement device 10 in the information processing device 40, the consistency of the SLAM process is maintained in the information processing device 40, making it possible to increase the reliability of the three-dimensional map to be created. In addition, not limited to the measurement device 10, image data obtained by other devices, etc. can also be used to generate and update the three-dimensional map, so that the three-dimensional map can be generated at a higher speed and with higher accuracy.

[0371] (Second Embodiment)

[0372] Fig.36 is a flowchart for explaining the process of SLAM processing according to the second embodiment. As Fig.36 shown, in the second embodiment, after obtaining the initial three-dimensional map in Fig.21 step S101, the first step is to transfer the initial three-dimensional map obtained in step S101 from the information processing device 40 to each measurement device 10 (step S141).

[0373] Next, each measurement device 10 acquires image data or feature points obtained by performing recognition processing on the image data, and 6DoF data acquired by the IMU 112 (step S142).

[0374] Subsequently, in each measurement device 10, based on the image data or feature points and 6DoF data acquired in step S142, the self-position on the three-dimensional map in the server map coordinate system is estimated (step S143), while the three-dimensional map is updated (step S144).

[0375] Next, the position information of each measurement device 10 in the server map coordinate system estimated in step S143 is sent from each measurement device 10 to the information processing device 40 (step S145), and thereafter, this operation proceeds to Fig.21 step S103.

[0376] In this way, according to the method of performing SLAM using a common initial three-dimensional map in each measurement device 10, the amount of data exchanged between each measurement device 10 and the information processing device 40 can be greatly reduced, resulting in faster body tracking of the user U. This makes it possible to improve the usability of the user U in operating an avatar in the virtual space.

[0377] (Third Embodiment)

[0378] Fig.37 is a flowchart for explaining the process of SLAM processing according to the third embodiment. As Fig.37 shown in Fig.21After obtaining the initial three-dimensional map in step S101, the first step is to generate a transformation matrix for converting the set of client map coordinate systems in each measurement device 10 into the server map coordinate system (refer to Fig.18 step S10 in

[0379] (step S151). Next, each measurement device 10 acquires image data or feature points obtained by performing recognition processing on the image data, and 6DoF data acquired by the IMU 112 (step S152).

[0380] Subsequently, in each measurement device 10, based on the image data or feature points and 6DoF data acquired in step S152, the self-position on the three-dimensional map in the client map coordinate system is estimated (step S153) to update the three-dimensional map (step S154).

[0381] Next, the position information of each measurement device 10 in the client map coordinate system estimated in step S153 is transmitted from each measurement device 10 to the information processing device 40 (step S155).

[0382] In response to this, the information processing device 40 uses the transformation matrix generated in step S151 to transform the position information received from each measurement device 10 into position information in the server map coordinate system (step S156), and thereafter, this operation proceeds to Fig.21 step S103 of

[0383] In this way, according to the method of performing SLAM using a separate three-dimensional map in each measurement device 10, compared with the second example, the amount of data transmitted and received between each measurement device 10 and the information processing device 40 can be further reduced, resulting in faster body tracking of the user U. This makes it possible to improve the usability of the user U in operating an avatar in the virtual space.

[0384] 1.6.5 Method of Inputting the Position of Each Measured Part into the Application and Operating the Human Model

[0385] Next, the method of operating the human model shown in step S104 of Fig.21 will be described. In step S104 of Fig.21 , the positions of each site (left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, head, etc.) of the user U measured in the server map coordinate system in step S103 are input into a predetermined application.

[0386] In this application, the shape of the human model displayed on the display device 50 changes according to the input position (three-dimensional coordinates) of each site. In other words, the three-dimensional position information obtained by measuring each part of the user U is used to operate the human model to be displayed on the display device 50. The process of changing the human model or the process of operating the human model can use known methods.

[0387] The human model displayed on the display device 50 can be, for example, a skeleton model associated with the three-dimensional position information of each part of the user U who measures the joints and body parts, or can be a three-dimensional shape model (e.g., a polygon model) in which the three-dimensional shape of a human body or an object imitating a human body is associated with the joints and body parts of the skeleton model, as Figure 3 shown. Alternatively, it can also be a three-dimensional model in which the skeleton model and the polygon model are overlapped and displayed.

[0388] As described above, the human body tracking system 1 according to the present embodiment uses a plurality of measurement devices 10 to measure the three-dimensional position information of each part of the human body, and converts the obtained position information of each part into the three-dimensional position information of each part of the human body in a common coordinate system in one of the measurement devices 10, the relay device 20, or the information processing device 40.

[0389] In the information processing device 40, the image processing unit 144 deforms the skeleton model according to the temporal change of the three-dimensional position information of each part of the human body (more specifically, the positions of the joints and segments in the skeleton model are moved, etc.), and outputs the deformed skeleton model. The image processing unit 144 further moves each vertex (corresponding to joints, etc.) of the polygon model according to the deformation of the skeleton model, and outputs the moved polygon model. In addition, the image processing unit 144 can also prepare a muscle model, obtain the shape change of the muscle according to the deformation of the skeleton model, move each vertex of the polygon model according to the shape change of the muscle, and output the moved model. Subsequently, the model output from the image processing unit 144 is output to the display device 50 connected to the information processing device 40 or the display device 50 integrated with the information processing device 40, and is displayed to the user U.

[0390] 1.7 Combined deformation of the units performing each process

[0391] Here, for rearrangement, in Fig.38 、 Fig.41 、 Fig.42 and Fig.46 a list of processes for implementing the above functions is shown, in Fig.39 、 Fig.42 、 Fig.45 and Fig.48 a list of the output data of each process is shown, and in Fig.40 、 Fig.43 , Fig.46 and Fig.49 show a list of combined changes in the units that perform each process. Note that, Figures 38 to 40 shows a situation where the position of the wrist is measured and reflected on a human model such as an avatar, Figure 41 to Figure 43 shows a situation where the positions of the elbow and shoulder are measured and reflected on a human model such as an avatar, Figures 44 to 46 shows a situation where the positions of the knee and ankle are measured and reflected on a human model such as an avatar, Figure 47 to Figure 49 shows a situation where the positions of the head and torso are measured and reflected on a human model such as an avatar.

[0392] As Figures 38 to 49 shown, the process from the process of measuring the position of each unit to the process of reflecting and displaying the measured position on the human model can be divided into multiple processes, and each separated process can be executed in any of the measuring device 10, the relay device 20, and the information processing device 40. Note that, Figures 38 to 49 the processes, output data, and combined variation examples shown in

[0393] 1.8 Variation Examples of System Configuration

[0394] Next, variations of the system configuration of the human body tracking system 1 according to the present embodiment will be described through some examples.

[0395] 1.8.1 First Variation Example

[0396] Fig.50 is a schematic diagram showing a schematic configuration example of a human body tracking system according to the first variation example. As Fig.50 shown, the human body tracking system 1A according to the first variation example can have a configuration without the communication device 30, the information processing device 40, and the display device 50, and includes a head-mounted display (HMD) 80 instead of these, which has these functions attached to the head of the user U.

[0397] 1.8.2 Second Variation Example

[0398] Fig.51 is a schematic diagram showing a schematic configuration example of a human body tracking system according to the second variation example. As Fig.51 shown, the second variation example uses a human body tracking system 1B having a configuration similar to that of the human body tracking system 1A according to the first variation example. However, in this configuration, the relay device 20 is omitted, and each measuring device 10 is configured to directly send data to the HMD 80 and receive data from the HMD 80. In this way, the relay device 20 can be omitted in the above-described embodiment and other variation examples, and is not limited to the second variation example.

[0399] 1.8.3 Third Modification Example

[0400] Fig.52 is a schematic diagram showing a schematic configuration example of a human body tracking system according to the third modification example. As Fig.52 shown, the third modification example can use a human body tracking system 1C including an HMD 80 instead of the display device 50, and the HMD 80 and the information processing device 40 communicate with each other via the communication device 30.

[0401] 1.8.4 Fourth Modification Example

[0402] Fig.53 is a schematic diagram showing a schematic configuration example of a human body tracking system according to the fourth modification example. As Fig.53 shown, the fourth modification example uses a human body tracking system 1D having a configuration in which the information processing device 40 is connected to a server (which can be a cloud or the like) 70 via a predetermined network 60. In such a configuration, part or all of the processing executed in the above-mentioned information processing device 40 can be executed in the server 70.

[0403] 1.8.5 Fifth Modification Example

[0404] Fig.54 is a schematic diagram showing a schematic configuration example of a human body tracking system according to the fifth modification example. As Fig.54 shown, in addition to the two measurement devices 10AL and 10AR attached to the two wrists of the user U, the fifth modification uses a human body tracking system 1E which also includes two measurement devices 10FL and 10FR attached to the two ankles. The two measurement devices 10FL and 10FR attached to the two ankles can have a configuration similar to that of the measurement devices 10AL and 10AR, and can estimate their own positions by SLAM and collect image data of each part of the user U. Such a configuration enables accurate SLAM estimation of the positions of the two ankles and the two wrists, resulting in more accurate body tracking.

[0405] 1.8.6 Sixth Modification Example

[0406] Fig.55 is a schematic diagram showing a schematic configuration example of a human body tracking system according to the sixth modification example. As Fig.55As shown, in the sixth modification example, a human body tracking system 1F is used, which has a configuration similar to the human body tracking system 1E according to the fifth modification example and further includes a measuring device 10H attached to the head of the user U. The measuring device 10H attached to the head may have a configuration similar to that of the other measuring devices 10, estimate its own position by SLAM, and acquire image data of each part of the user U. Such a configuration enables accurate SLAM estimation of the position of the head in addition to both the wrists and ankles, resulting in body tracking with higher accuracy.

[0407] 1.9 Overview

[0408] As described above, according to the present embodiment and its modification examples, when there is free movement in the positions of the measuring devices 10AL and 10AR attached to the user U (e.g., attached to the wrists), their positions can be estimated and the positions of each part of the user U can be measured based on the estimated positions and image data or distance measurement results. Regarding the number of parts of the user U as the measurement object, the number of parts that can be estimated is more than the number of measuring devices 10. The positions of a large number of parts on the user U can be measured by a small number of measuring devices 10. For example, n measuring devices 10 can detect the postures and movements of N parts, where N is more than n. The problem of wearing discomfort caused by conventionally attaching a large device or multiple devices to the user U and the hindrance of user movement occurring in the user U can be suppressed. Natural postures and movements can also be detected.

[0409] 2. Second Embodiment

[0410] The second embodiment is different from the first embodiment in the position estimation method of each measuring device 10 (e.g., the measuring devices 10AL and 10AR). In the second embodiment, the measuring devices 10AL and 10AR estimate their own positions by capturing images of markers (described in detail below) worn by the user U. Different from the first embodiment, it is not necessary for the measuring devices 10AL and 10AR to perform SLAM. Similar to the first embodiment, for example, after position estimation, the positions of each part of the user U are triangulated using the measuring devices 10AL and 10AR to measure the positions of each part of the user U. Since the measuring devices 10AL and 10AR do not need to use SLAM technology, only a small amount of calculation is required to estimate their positions and measure the positions of each part of the user U. The processing volume of the entire system can also be reduced.

[0411] The marker in the second embodiment is denoted as marker MZ to distinguish it from marker MK in the first embodiment described above. Marker MZ can be any marker as long as feature points can be extracted when the measurement devices 10AL and 10AR image marker MZ. For example, marker MZ can represent a marker with a known size that is fixed within user U. Marker MZ can be a marker worn by user U, or can be a marker having a body surface portion of user U as a feature point. More specifically, the position of the body surface portion can be measured from the outside, and its shape is not deformed by the posture of user U. Examples of markers MZ that can be worn by user U include the wearing of, for example, a bib, a board, a belt, a window frame, or a patterned portion (pattern, etc.) of a shirt. Examples of body surface portions of user U are portions such as the collarbone and ribs.

[0412] 2.1 System configuration example

[0413] Fig.56 FIG. is a schematic diagram showing a schematic configuration example of a human body tracking system according to the second embodiment. Similar to the first embodiment described above, the measurement device 10AL is attached to the left wrist of user U, and the measurement device 10AR is attached to the right wrist of user U. Compared with the first embodiment, the human body tracking system 2 further includes a marker MZ. In this embodiment, the marker MZ is fixed to the trunk of user U.

[0414] The measurement devices 10AL and 10AR separately capture an image of the marker MZ using the sensor 111 ( Figure 2 ). Based on the imaging result, more specifically, for example, the image data, the respective positions of the measurement devices 10AL and 10AR are estimated. The process for estimating the position is executed by the processing unit 113, and its specific method will be described below. For example, similar to the first embodiment, based on the estimated positions of the measurement device 10AL and the measurement device 10AR, triangulation is used to measure the position of each part of user U.

[0415] A method for estimating the positions of the measurement devices 10AL and 10AR will be described. The measurement devices 10AL and 10AR attached to the two wrists of user U separately image the marker MZ. Based on the image data obtained in this way, the positions and postures (orientations) of the measurement devices 10AL and 10AR are estimated separately. In Fig.56 the example shown, the measurement devices 10AL and 10AR are attached to the wrists of user U, and thus, the positions and postures of the measurement devices 10AL and 10AR are estimated to measure the positions of the wrists of user U. A method for measuring the position of each part of user U including the wrists will be described.

[0416] 2.2 Method for measuring the position of each part

[0417] 2.2.1 Wrist

[0418] As described above, estimating the positions and postures of the measurement devices 10AL and 10AR measures the position of the wrist of the user U. This will also be described with reference to Fig.57 description.

[0419] Fig.57 is a diagram for showing the estimation of the positions and postures of the measurement devices. The user U wears the measurement devices 10AL and 10AR on their left wrist and right wrist, respectively. Each dotted circle shown around the measurement devices 10AL and 10AR represents a detection range, where, similar to Figure 7 , each measurement device 10 can detect an object, and basically represents that the detection range is 360° around each measurement device. Thus, the measurement device 10AL and the measurement device 10AR can image the marker MZ attached to the trunk of the user U. As will be described later, based on the image data of the feature points including the marker MZ, the positions of the measurement devices 10AL and 10AR (positions in the world coordinate system based on the marker MZ) are estimated. Since the measurement device 10 is attached to the wrist, the estimated positions of the measurement device 10AL and the measurement device 10AR are regarded as the positions of the wrist to be measured.

[0420] In the present embodiment, the measurement device 10AL and the measurement device 10AR respectively capture images of the marker MZ and acquire the image data of the marker MZ. Since the image data of the marker MZ with a known size has been obtained, the positions and postures of the sensors 111 (e.g., cameras) in each of the measurement devices 10AL and 10AR, and finally the positions and postures of each of the measurement devices 10AL and 10AR, can be obtained by solving the perspective-n-point problem (PnP problem). Specifically, for example, using the determinant in the following formula (4), the relationship between (i) the position of the marker MZ in the world coordinate system (three-dimensional coordinates, world coordinate points), (ii) the position of the marker MZ in the captured image (two-dimensional coordinates, image coordinate points), (iii) the external parameters of the camera, and (iv) the internal parameters of the camera is described.

[0421]

[0422] (i) Since the coordinates of the feature points P1 to P4 of the marker MZ in the world coordinate system are known, the position of the marker MZ in the world coordinate system can be obtained. (ii) The measuring devices 10AL and 10AR respectively capture images of the marker MZ and obtain the position of the marker MZ in the captured images. (iv) The internal parameters are values specific to the imaging camera and are thus known. Therefore, the external parameters (translation vector t and rotation vector r) of (iii) can be obtained by calculation. This makes it possible to obtain, that is, when the marker MZ is imaged, the position and orientation of the camera (the direction of the optical axis of the lens provided in the camera) can be estimated. The positions and orientations of the measuring devices 10AL and 10AR can also be estimated. Reference will also be made to Figure 57 to Figure 59 for a description thereof.

[0423] Fig.58 is a flowchart showing an example of a process for estimating the position and posture of a measuring device. Fig.59 is a view showing an example of a marker (e.g., a bib or a board) attached to a user, Fig.60 is a view showing an example of a world coordinate system set with the marker as a reference.

[0424] In this example, for example, as Fig.59 shown, the marker MZ is attached to the trunk of the user U. The four corners of the marker MZ can be the feature points P1 to P4 for specifying the shape, area, extension plane, etc. of the marker MZ.

[0425] As Fig.58 shown, the first step of this operation is to set a reference coordinate system (world coordinate system) based on the marker MZ attached to the trunk of the user U (step S201). For example, as Fig.60 shown, the origin o of the world coordinate system is located within the region including the marker MZ to set the world coordinate system, where the horizontal direction on the extension surface of the marker MZ is defined as the x direction, the vertical direction is defined as the y direction, and the vertical direction with respect to the extension plane of the marker MZ is defined as the z direction.

[0426] Next, the positions (three-dimensional coordinates) of the feature points P1 to P4 of the marker MZ are determined on the world coordinate system (step S202). In other words, the size of the marker MZ and the positional relationship of the plurality of feature points (in this example, the feature points P1 to P4) included in the marker MZ are known, and thus this information is used to represent the positions of the plurality of feature points included in the marker MZ in the world coordinate system. This process corresponds to the process of obtaining the third term on the right side of the above formula (4). It should be noted that in the case of this embodiment, since the feature points P1 to P4 of the bib MZ exist in the extension plane of the marker MZ, the z coordinates of the feature points P1 to P4 in the world coordinate system are "0".

[0427] The next step is to determine the positions of the feature points P1 to P4 of the marker MZ in the camera coordinate system. Specifically, the image data of the marker MZ obtained by each of the measuring devices 10AL and 10AR is used to obtain the positions of the feature points P1 to P4 of the marker MZ on the two-dimensional coordinate system (image coordinate system) of the captured image (step S203). This process corresponds to the process of obtaining the left side of the above formula (4). The process of extracting the feature points P1 to P4 from the image data can use, for example, a process of extracting a target region in units of pixels by semantic segmentation and detecting vertices in the extracted region.

[0428] Next, for example, in the case where three corresponding points have been obtained, the external parameters can be estimated by solving a non-linear equation. This results in the estimation of the camera position, that is, the positions of the measuring devices 10AL and 10AR (step S204). The external parameters are vectors at the time of imaging the marker MZ, including: a translation vector t from the origin to the camera position; and a rotation vector r for rotating the orientation of the camera in the world coordinate system. These vectors are regarded as vectors representing the degree and direction of translation of the origin of the local coordinate system (camera coordinate system and client coordinate system) with the camera as the origin of the world coordinate system and representing the degree and direction of rotation of the coordinate axes of the local coordinate system from the coordinate axes of the world coordinate system.

[0429] As described in the first embodiment, obtaining three or more corresponding feature points enables the integration of the camera coordinate system and the world coordinates defined and limited by the marker MZ and the like. That is, the coordinate system is integrated across the measuring devices 10AL and 10AR attached to the left and right wrists of the user U. In other words, separate self-positions are obtained on a similar world coordinate system as the self-positions of the measuring devices 10AL and 10AR. This will also be described with reference to Fig.61 be described.

[0430] Fig.61This is a diagram for showing coordinate integration. The measurement devices 10AL and 10AR have a basic 360° FOV capability and continuously image each body part of the marker MZ and the user U. The measurement device 10AL determines the position of the marker MZ included in the captured image in the image coordinate system, and solves the above PnP problem based on this position. Thus, the self-position of the measurement device 10AL in the reference coordinate system (world coordinate system) based on the marker MZ is always determined. Similarly, the measurement device 10AR obtains the position of the marker MZ included in the captured image in the image coordinate system, and solves the PnP problem based on this position. Thus, the self-position of the measurement device 10AR in the reference coordinate system (world coordinate system) based on the marker MZ is always obtained. That is, both the measurement device 10AL and the measurement device 10AR image the same marker MZ, and continuously obtain their self-positions in the reference coordinate system (world coordinate system) based on the same marker MZ. In other words, separate self-positions are continuously obtained on a similar world coordinate system as the self-positions of the measurement devices 10AL and 10AR.

[0431] There are cases where the position of the marker MZ moves or does not move. First, the case where the position of the marker MZ does not move will be described. For example, a conceivable application is an application for performing human body tracking in a state where the user U does not move or the main body part to which the marker MZ is attached does not move. The position of the marker MZ is fixed and thus known. When the user U moves the wrist, the self-positions of the measurement devices 10AL and 10AR are estimated in order to track the movement. Specifically, as described above, based on the feature points of the marker MZ in the image data acquired by the measurement device 10AL, the PnP problem is solved, and thus the self-positions of the measurement devices 10AL and 10AR are estimated. In other words, the positions and their orientations (e.g., the orientation of the optical axis of the lens) of the measurement devices 10AL and 10AR in the world coordinate system are obtained. Each part of the user U is captured by the measurement devices 10AL and 10AR. In other words, images of each part of the user U are captured by the measurement devices 10AL and 10AR. In addition, the positions of the measurement devices 10AL and 10AR in the world coordinate system and their orientations (e.g., the orientation of the optical axis of the lens) when capturing each part of the user U in the image have been obtained. Using the above information, triangulation is used to measure the positions of each part of the user U captured by the measurement devices 10AL and 10AR.

[0432] Next, the case where the position of the marker MZ is moved will be described. For example, an imaginable application is an application that performs body tracking in a state where the user U moves or changes in the posture of the user U, including the trunk portion to which the marker MZ is attached. In this case, it is permissible to attach the measurement device 10 (measurement device 10B described below) to the user U in a state where the positional relationship with the marker MZ does not change. Only this single measurement device can use the SLAM technology, thereby (known) grasping the position of the marker MZ that moves together with the measurement device. Therefore, even when the marker MZ moves, the respective self-positions of the measurement devices 10AL and 10AR can be estimated. This will also be described with reference to Fig.62 be described.

[0433] Fig.62 is a schematic diagram showing a schematic configuration example of a human body tracking system. In this example, the human body tracking system 2 further includes a measurement device 10B attached to the trunk of the user U. The functional configuration of the measurement device 10B can be, for example, similar to the functional configuration of the measurement device 10 described above with reference to Figure 2 etc. in the first embodiment. The measurement device 10B can be attached to the user U so as to have no positional relationship with respect to the marker MZ. For example, the measurement device 10B and the marker MZ can be fixed to the same rigid member.

[0434] The measurement device 10B can be configured to be capable of performing SLAM, for example, and the self-position of the measurement device 10B on the three-dimensional map can be estimated by SLAM. The processing performed by SLAM can be similar to the processing of the first embodiment or its modifications described above, and thus its description will be omitted here. The measurement device 10B and the marker MZ are fixed to suppress changes in their positional relationship. Therefore, even if the position of the marker MZ moves, by obtaining the position of the measurement device 10B, the position of the marker MZ can be grasped. In addition, similar to the case where the marker MZ does not move as described above, the positions of the measurement devices 10AL and 10AR relative to the position of the moved marker MZ can be grasped. The positions of the measurement devices 10AL and 10AR relative to the position of the moved marker MZ can be grasped, and both the position of the marker MZ before movement and the position of the marker MZ after movement can be grasped. As a result, even when the marker MZ moves, the positions of the measurement devices 10AL and 10AR relative to the position of the marker MZ before movement can be grasped. That is, even when the marker MZ moves, the positions of the measurement devices 10AL and 10AR in the world coordinate system before the marker MZ moves can be grasped. Subsequently, by capturing each part of the user U using both the measurement device 10AL and the measurement device 10AR whose positions are obtained in the world coordinate system, in other words, by capturing images of each part of the user U using both the measurement device 10AL and the measurement device 10AR, the position of each part of the user U captured by both the measurement device 10AL and the measurement device 10AR is measured using triangulation based on the information obtained by imaging.

[0435] The positions of the two wrists of the user U can be measured by estimating the positions of the measurement devices 10AL and 10AR. Similar to the first embodiment and its variations, techniques such as triangulation can be used to measure the positions of each part of the user U other than the wrists. Unnecessary explanations are omitted as needed.

[0436] 2.2.2 Elbows and Shoulders

[0437] As described above, the positions of the two wrists of the user U can be estimated and finally measured based on the image data including the feature points of the bib MZ individually acquired by the measurement devices 10AL and 10AR. Therefore, a method similar to the method described in referring to Figure 8 in the first embodiment can be used to measure the positions of the elbows and two shoulders of the user U.

[0438] 2.2.3 Knees and Ankles

[0439] The method for measuring the positions of the knees and ankles can be similar to the measurement method described in the first embodiment of reference Fig. 9 described above.

[0440] 2.2.4 Head and torso

[0441] The method of measuring the positions of the head and the trunk can be similar to the measurement method described in the first embodiment above with reference to Fig.11 the measurement method described.

[0442] 2.2.5 Another method of measuring the position of each site

[0443] Similar to the first embodiment above, at least one of the at least one sensors 111 in the measurement device 10AL and the measurement device 10AR can be a distance measurement sensor. By using the depth data acquired by the sensor 111, the position of each part of the user U relative to the measurement devices 10AL and 10AR can be measured. The position measurement using the distance measurement sensor is as described in reference Fig.12 in the first embodiment.

[0444] 2.2.6 Method of measuring the shape of the hand / finger

[0445] The method of measuring the shape of the fingers of the user U can be similar to the measurement method described in the first embodiment above with reference to Figures 13 to 16 the measurement method described.

[0446] 2.3 Coordinate system

[0447] Next, the process of transforming the camera coordinate system of each measurement device 10 worn by the user U will be described through some examples. In Figure 63 to Figure 65 , the step on the starting side of the arrow indicates the coordinate system before transformation, and the step on the front side of the arrow indicates the coordinate system after transformation. In addition, the camera coordinate system of the measurement device 10B attached to the trunk of the user U to be distinguished from other camera coordinate systems is represented as the breast camera coordinate system.

[0448] (First embodiment)

[0449] The first example will be described as the process of transforming coordinates to the camera coordinate system of each measurement device 10 attached to each part via the measurement device 10 (for example, the measurement device 10B) that images the outside of the user U using the external coordinates as the absolute reference.

[0450] Fig.63 is a schematic diagram for showing the transformation process according to the first embodiment. As Fig.63 shown in, in the first example, first, the three-dimensional coordinates of the external coordinate system are transformed into the three-dimensional coordinates of the breast camera coordinate system (steps S211 to S212). The external coordinate system can be a coordinate system having a specific position in the real space (for example, the position of a marker fixed to the external environment) as the origin.

[0451] Next, the three-dimensional coordinates in the breast camera coordinate system are transformed into the camera coordinate system of the measurement device 10 attached to each part of the user U (in this example, the two wrists) (Steps S212 to S213).

[0452] In this procedure, the above-mentioned world coordinate system may be an external coordinate system or a breast camera coordinate system.

[0453] (Second Embodiment)

[0454] The second example will be described as a process of directly transforming coordinates into the camera coordinates of the measurement device 10 attached to each part of the user U using the external coordinates as an absolute reference. In this case, the measurement device 10 for each part needs to capture an image of a reference (e.g., a marker, etc.) fixed in the real space.

[0455] Fig.64 is a diagram for showing the transformation process according to the second embodiment. As Fig.64 shown, in the second embodiment, the three-dimensional coordinates of the external coordinate system are directly transformed into the camera coordinate system of the measurement device 10 attached to each part of the user U (in this embodiment, the two wrists) (Steps S221 to S222).

[0456] In this procedure, the above-mentioned world coordinate system may be an external coordinate system.

[0457] (Third Embodiment)

[0458] The third embodiment will be described as a process of using coordinate transformation to transform the camera coordinates of the measurement device 10 attached to each part of the user U (in this embodiment, the two wrists) using the relative coordinates from the measurement device 10B attached to the torso. In this case, it is impossible to measure the absolute coordinates of each part to which the measurement device 10 is attached, and therefore, this process is desirable for applications that perform body tracking in the case where there is no large movement at the center of the body of the user U.

[0459] Fig.65 is a diagram for showing the transformation process according to the third embodiment. As Fig.65 shown, in the third embodiment, the three-dimensional coordinates of the breast camera coordinate system are directly transformed into the camera coordinate system of the measurement device 10 attached to each part of the user U (in this embodiment, the two wrists) (Steps S231 to S232).

[0460] In this procedure, the above-mentioned world coordinate system may be a breast camera coordinate system.

[0461] In the present embodiment, there are coordinates when the measurement device 10AL or the measurement device 10AR images the front side (front surface) of the user U's body and coordinates when imaging the rear side (rear surface) of the user U's body, and it is necessary to grasp the relationship between these situations. For example, in the case where the user U wearing the measurement devices 10AL and 10AR on the left wrist and the right wrist runs, one wrist is on the front side of the body and the other wrist is on the rear side of the body. The sensors 111 (e.g., cameras) of each of the measurement devices 10AL and 10AR have a 360° FOV. Therefore, even when the user U's wrist is on the front side or the rear side of the body, each of the measurement devices 10AL and 10AR can continuously image the left foot and the right foot of the user U. However, the marker MZ attached to the front side of the user U's body cannot be imaged from the rear side of the user U's body. Therefore, the marker MZ is also attached to the back of the user U's body so that the marker MZ can be imaged from the back of the user U's body.

[0462] For example, there are conceivable situations where one of the measurement devices 10AL and 10AR attached to the left wrist and the right wrist of the user U captures an image of the front side of the user U's body and the other captures an image of the rear side of the user U's body, or a situation where one of the measurement devices moves from the front side to the rear side of the user U's body. The respective camera coordinates of the measurement device 10AL and the measurement device 10AR include the coordinates when viewing (capturing) the marker MZ from the front side of the user U's body and the coordinates when viewing the marker MZ from the rear side of the user U's body. These two coordinates need to be integrated into one coordinate system by, for example, a method of transforming the coordinate system into one of the two coordinate systems. Conceivable methods for integrating the camera coordinate systems include the following two methods.

[0463] The first method will be described as an exemplary case where the user U wears a rigid member such as a belt or a protective device extending from the front side to the rear side of the object. The rigid member should be set as a member extending from the front side to the rear side of the user U's body and having a known shape. In this case, by placing the first marker MZ for the front side of the body on the front side and the second marker MZ for the rear side of the body on the rear side, the positional relationship between the first marker MZ for the front side and the second marker MZ for the rear side is constantly fixed and known (can be measured in advance). As a result, for example, in the case where the wrist and the measurement device 10AL or the measurement device 10AR are on the rear side of the body, the position of the measurement device 10AL or the measurement device 10AR can be transformed, which is obtained by imaging the second marker MZ on the back side of the body and is the position represented in the second world coordinate system having its origin in the area of the marker MZ on the back side of the body, in the coordinates represented in the first world coordinate system having its origin within the area of the marker MZ on the front side of the body.

[0464] A second method will be described. In an imaginable case, the first marker MZ is fixed to the front side (chest side) of the user U, and the second marker MZ is fixed to the back side (rear side) of the user U. The positional relationship between the first marker MZ and the second marker MZ is unknown. Taking the measuring device 10AL as an example, the first marker MZ is imaged once from the front side of the user U through the wrist and the measuring device 10AL attached thereto, and based on this, the self-position of the measuring device 10AL when the first marker MZ is imaged is obtained, and this self-position is represented in the first world coordinate system with the origin in the area of the first marker MZ. Along with this, the second marker is imaged once from the back side of the user U through the wrist and the measuring device 10AL attached to the wrist, and based on this, the self-position of the measuring device 10AL when the second marker is imaged is obtained, and this self-position is represented in the second world coordinate system with the origin in the area of the second marker MZ. In addition, by measuring the change in the position and orientation of the measuring device 10AL from the point of imaging the first marker MZ to the point of imaging the second marker MZ by the IMU 112 included in the measuring device 10AL, the relationship between the specific position represented in the first world coordinate system (the position of the measuring device 10AL when imaging the first marker MZ) and the specific position represented in the second world coordinate system (the position of the measuring device 10AL when imaging the second marker MZ) can be grasped. Once this relationship is grasped, all the coordinates of the measuring device 10AL obtained by imaging the second marker MZ and now represented in the second world coordinate system can be freely represented by the coordinates represented in the first world coordinate system. A similar application is made to the imaging using the measuring device 10AR.

[0465] 2.4 Combinatorial deformation of the units performing each process

[0466] Here, for rearrangement, a list of processes for implementing the above functions is shown in Fig.66 、 Fig.69 、 Fig.72 and Fig.75 , a list of output data of each process is shown in Fig.67 、 Fig.70 、 Fig.73 and Fig.76 , and a list of combinatorial changes of the units performing each process is shown in Fig.68 、 Fig.71 、 Fig.74 and Fig.77 . Note that Figure 66 to Figure 68 shows the case where the position of the wrist is measured and reflected on a human model such as an avatar, Figure 69 to Figure 71 shows the case where the positions of the elbow and shoulder are measured and reflected on a human model such as an avatar, Figure 72 to Figure 74Illustrates the situation where the positions of the knees and ankles are measured and reflected on a human model such as an avatar. Figure 75 to Figure 77 Illustrates the situation where the positions of the head and torso are measured and reflected on a human model such as an avatar.

[0467] As Figures 66 to 77 shown, the process from the process of measuring the position of each unit to the process of reflecting and displaying the measured position on the human model can be divided into multiple processes, and each separated process can be executed in any of the measuring device 10, the relay device 20, and the information processing device 40. Note that the processes, output data, and combined variant examples shown in Figures 66 to 77 are embodiments and are not limited to the above.

[0468] 2.5 Variations in System Configuration

[0469] Next, variations in the system configuration of the human body tracking system 2 according to the present embodiment will be described through some examples.

[0470] 2.5.1 First Variant Example

[0471] Fig.78 is a schematic diagram showing an exemplary configuration of a human body tracking system according to the first variant example. Fig.79 is a block diagram showing an exemplary configuration of a relay device according to the first variant example. In the above second embodiment, the measuring device 10B is provided as a device for detecting the movement of the body and the change in the posture of the user U. In the first variant example, the relay device 220 includes an IMU 221 and a GNSS receiver 222 as devices for detecting the movement of the body and the change in the posture of the user U, and uses these units to detect the movement of the body and the change in the posture of the user U.

[0472] As Fig.78 shown, the human body tracking system 2A according to the first variant example has a configuration similar to the above Fig.62 . However, in this configuration, the measuring device 10B attached to the backbone of the user U is omitted, and the relay device 20 attached to the user U is replaced by the relay device 220.

[0473] The positional relationship between the marker MZ and the relay device 220 is maintained in a constant positional relationship. For example, both the relay device 220 and the marker MZ can be attached to the body surface of the user U, which is a body surface with an immutable shape (e.g., the chest). The user U can wear a rigid member and fix the relay device 220 and the marker to the rigid member. The relay device 220 attached to the body surface of the user U (e.g., the chest) as a body surface with an immutable shape can also be used as the marker MZ. The user U can wear a fixing device that fixes an object to the body of the user U and fix the relay device 220 to the fixing device so that the fixed relay device 220 can be used as the marker MZ.

[0474] As Fig.79 shown, in addition to the configuration similar to that of the relay device 20 described with reference to Figure 2 the first embodiment, the relay device 220 further includes an IMU 221 and a Global Navigation Satellite System (GNSS) receiver 222.

[0475] That is, in the first modification, the three-dimensional position of the relay device 220 is specified based on the GNSS signal received by the GNSS receiver 222, and the posture of the relay device 220 is estimated based on the 6DoF data detected by the IMU 221. In the first modification, the world coordinate system is set with the relay device 220 as a reference. For example, it is possible to set the world coordinate system in which the position of the relay device 220 is the origin, the front direction of the relay device 220 is the z direction, the horizontal direction parallel to the front is the x direction, and the vertical direction parallel to the front is the y direction.

[0476] By setting the world coordinate system in this way using the GNSS signal and the IMU 221, the position estimation using SLAM can be omitted, thereby further reducing the processing amount of the entire system.

[0477] In the first modification, for example, three or more feature points (e.g., the corners of the relay device 220, etc.) in the relay device 220 can be used to integrate the camera coordinate system and the world coordinate system of each measuring device 10.

[0478] Compared with the second embodiment including the measuring device 10B, the first modification uses the IMU 221 and GNSS as means for detecting the body movement and the posture change of the user U. Compared with the second embodiment in which SLAM is performed by imaging the outside world as the detection of the body movement and the change of the posture of the user U, the first modification requires less computational amount for detecting the body movement and the change of the posture of the user U.

[0479] 2.5.2 Second modification

[0480] Fig.80This is a schematic diagram showing an exemplary configuration of a human body tracking system according to a second modification. As Fig.80 shown, the human body tracking system 2B according to the second modification has a configuration similar to that Fig.62 described above. However, in this configuration, the measurement device 10B attached to the torso of the user U is changed to a measurement device 10H attached to the head of the user U.

[0481] In this way, when the measurement device 10H serving as a reference is mounted on the head, a marker MZ for setting the world coordinate system is provided at a position on the head where the positional relationship with the measurement device 10H does not change. Examples of the marker MZ may include characteristic positions on the head of the user U, such as the eyes, nose, mouth, ears, or chin of the user U, or characteristic shapes or patterns worn or rendered on the user U, such as glasses, a hat (cap), a headband, or a painted picture.

[0482] Note that the attachment part of the measurement device 10 serving as a reference is not limited to the torso and the head, and may be another part such as the wrist and the ankle. In this case, the marker MZ for setting the world standard (reference) may be provided at a part where the positional relationship with the measurement device 10 serving as the standard (reference) does not change.

[0483] 2.6 Overview

[0484] As described above, according to the present embodiment and its modifications, the self-positions of the measurement devices 10AL and 10AR can be estimated with a specific marker MZ as a reference. Therefore, similar to the first embodiment and its modification examples described above, even when the positions of the measurement devices 10AL and 10AR move freely, the positions of each part of the user U can be measured. In addition, it is not always necessary to use SLAM in the measurement devices 10AL and 10AR. In the absence of SLAM, it is more likely that the processing amount of the entire system can be reduced.

[0485] 3. Third Embodiment

[0486] The third embodiment is different from the first and second embodiments in the position estimation method of each measurement device 10 (for example, the measurement devices 10AL and 10AR). In the third embodiment, for example, another measurement device 10 attached to the user U is used to measure the positions of the measurement devices 10AL and 10AR.

[0487] 3.1 Exemplary System Configuration

[0488] Fig.81 This is a schematic diagram showing an exemplary configuration of a human body tracking system as an information processing system according to the present embodiment. Fig.82It is a block diagram showing a schematic configuration example of a measurement device attached to the head according to the present embodiment.

[0489] As Fig.81 shown, in addition to the configuration similar to that of the human body tracking system 1 described with reference to Figure 1 the first embodiment, the human body tracking system 3 further includes a reference measurement device 310H attached to the head (or trunk) of the user U.

[0490] In the present embodiment, the measurement devices 10 attached to parts other than the head (in this example, the measurement device AL and the measurement device 10AR) have the function of transmitting a positioning signal (such as a beacon) to the reference measurement device 310H. The signal can be, for example, a beacon in Bluetooth or the like. In this case, the signal transmitter can be implemented by using the communication unit 115 included in the measurement device 10. In the following description, for the sake of clarity, the signal used for positioning in Bluetooth is represented as a beacon.

[0491] For example, as Fig.82 shown, the reference measurement device 310H has a configuration similar to that of the measurement device 10, receives the beacons sent from each measurement device 10, and measures the approximate positions of each measurement device 10.

[0492] A plurality of sensors 111 in the reference measurement device 310H are each fixed at a predetermined position on the outer surface of the housing of the reference measurement device 310H so as to be able to image the entire circumference of the user U and capture the measurement devices 10 attached to other parts (in this example, two wrists) by two or more of the sensors 111. Each sensor 111 can be a sensor such as an image sensor (including an IR sensor), a distance measuring sensor, an EVS, a hybrid sensor, similar to the above-described embodiment or its modification.

[0493] In such a configuration, the reference measurement device 310H uses the beacons sent from the measurement devices 10 as indicators (that is, the measurement devices 10AL and 10AR attached to the two wrists as clues), and uses two or more sensors 111 to capture images of each measurement device 10. The position of each measurement device 10 relative to the reference measurement device 310H imaged by two or more sensors 111 can be measured by triangulation using the image data acquired by two or more sensors 111 of the reference measurement device 310H.

[0494] It should be noted that in the case where there is a measurement device 10 not captured by two or more sensors 111 of the reference measurement device 310H, the 6DoF data detected by the IMU 112 included in the measurement device 10 can be used to measure the position of the measurement device 10.

[0495] The position of the place equipped with the reference measurement device 310H can be estimated using, for example, 6DoF data detected by the IMU 112, can be estimated using SLAM, etc., or can be estimated by the provided GNSS receiver 311 as shown in Fig.82 .

[0496] Similar to the above-described embodiment or its modification, triangulation or the like can be used to measure other parts.

[0497] 3.1.1 Variation of System Configuration

[0498] As in the Fig.83 reference measurement device 310HD in the human body tracking system 3A shown in, the reference measurement device 310H according to the present embodiment may include at least one function of the relay device 20, the communication device 30, the information processing device 40, and the display device 50. When equipped with the function of the display device 50, the reference measurement device 310HD can be provided in the form of an HMD.

[0499] 3.2 Configuration Embodiment of the Measurement Device Attached to the Head

[0500] Fig.84 is an external view showing a schematic configuration example of the measurement device attached to the user's head according to the present embodiment, where (A) shows a top view of the reference measurement device 310H, (B) shows a left view of the reference measurement device 310H, and (C) shows a front view of the reference measurement device 310H.

[0501] As Fig.84 shown, the reference measurement device 310H has a configuration including a helmet-shaped housing and having a plurality of sensors 111 placed around the circumference of the helmet-shaped housing. The housing is not limited to the helmet shape and can be differently modified into a shape such as a glasses shape or a headband shape, for example, as long as the housing has a shape that enables the plurality of sensors 111 to be placed around the head of the user U.

[0502] Figures 85 to 88 are diagrams each showing an example of the arrangement of sensors in the measurement device attached to the user's head according to the present embodiment and the FOV of each sensor. Figures 85 to 87 shows an embodiment of the arrangement and FOV of the sensors 111 when the reference measurement device 310H is viewed from the top, and Figure 88 shows an embodiment of the arrangement and FOV of the sensors 111 when the reference measurement device 310H shown in Figure 87 or Figure 84 is viewed from the side.

[0503] As Figures 85 to 87 andFigure 88 As shown in Figure 88 , the reference measurement device 310H has a configuration with a plurality of sensors 111 arranged on the periphery of the housing (i.e., the periphery of the head of the user U) such that the circumference of the head of the user U falls within the viewing angles of at least two sensors 111. Figure 85 An example configuration is shown in which one of the four sensors arranged substantially uniformly around the housing is arranged to face the front of the user U. Figure 86 An example configuration is shown in which the four sensors arranged substantially uniformly around the housing are arranged to face the diagonal direction of the user U, and Figure 87 An example configuration is shown in which eight sensors 111 are arranged substantially uniformly around the housing.

[0504] In such a configuration embodiment, the field of view (FOV) angle of the lens included in each sensor 111 is preferably 180 degrees or greater, but is not limited thereto and may be less than 180 degrees as long as the circumference of the head of the user U can be maintained within the viewing angles of at least two sensors 111.

[0505] During a period when the part equipped with the measurement device 10 (such as the wrist) is in an area that cannot be captured by at least two sensors 111 in the reference measurement device 310H, the 6DoF data detected by the IMU 112 included in the measurement device 10 can be used to measure the position of the corresponding part.

[0506] 3.3 Position measurement method for each part

[0507] Next, a method for measuring the position of each part in the user U according to the present embodiment will be described in detail with reference to the drawings. In the present embodiment, similar to the above-described embodiment and its variations, the position measurement parts are represented by the left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, and head, but the position measurement parts are not limited thereto, and various parts in the user U can be defined as the position measurement objects. In addition, in the following description, for simplicity, the right and left sides will not be distinguished as needed.

[0508] 3.3.1 Head and wrists

[0509] First, a method for measuring the positions of the head and two wrists of the user U will be described. Figure 89 is a schematic diagram for showing a method for measuring the positions of the head and wrists according to the present embodiment.

[0510] In the present embodiment, the position of the head of the user U can be measured by using SLAM performed by the reference measurement device 310H or the IMU 112 or GNSS receiver 311 included in the reference measurement device 310H.

[0511] The multiple sensors 111 fixed to the housing of the reference measurement device 310H have known mutual positional relationships. That is, there is known information about the position of each sensor 111 in the client coordinate system or the server map coordinate system of the reference measurement device 310H. Therefore, for example, based on the image data obtained by at least two sensors 111 capturing the wrist and the positional relationship between the two sensors 111, the position of each wrist of the user U can be measured by a method such as triangulation.

[0512] At that time, the reference measurement device 310H can specify at least two sensors 111 that capture at least one of the two wrists of the user within the viewing angle by using the beacons emitted by the measurement devices 10 attached to each of the two wrists of the user U as indicators.

[0513] 3.3.2 Elbows and Shoulders

[0514] As described above, the positions of the two wrists of the user U can be obtained from the image data acquired by at least two sensors 111 in the reference measurement device 310H. Therefore, a method similar to the method described in Figure 8 the first embodiment can be used to measure the positions of the elbows and two shoulders of the user U.

[0515] 3.3.3 Knees, Ankles, and Torso

[0516] Next, a method for measuring the positions of the two knees, two ankles, and the torso of the user U will be described. Figure 90 is a diagram for explaining the method for measuring the knees, ankles, and torso of the present embodiment.

[0517] As Figure 90 shown, also in the present embodiment, the two knees, two ankles, and the torso of the user U have been captured by the two measurement devices 10 (i.e., the measurement device 10AL and the measurement device 10AR) attached to the two wrists of the user U. Subsequently, the position of each measurement device 10 (in this example, the wrist position) in the client coordinate system or the server map coordinate system is measured by the above method. Therefore, based on the image data acquired by the two measurement devices 10 capturing the knees, ankles, and torso and the positional relationship between the two measurement devices 10, the positions of the two knees, two ankles, and the torso of the user U can be measured by a method such as triangulation.

[0518] 3.3.4 Another Method for Measuring the Position of Each Site

[0519] Although the above measurement method is an exemplary case of measuring the position of each part of the user U by triangulation from the image data imaged by two measurement devices 10 with known self-positions, the measurement method according to the present embodiment is not limited thereto, and various modifications can be made. For example, as mentioned in the first embodiment, when at least one of the at least one sensor 111 in the measurement device 10 with known self-position is a distance measurement sensor, the position of each part of the user U relative to the measurement device 10 with known self-position can be measured by using the depth data obtained by the sensor 111.

[0520] In the case of the present embodiment, the self-positions of the measurement device 10AL attached to the left wrist and the measurement device 10AR attached to the right wrist have been measured by the above method. Therefore, by using at least one of the sensors 111 included in the measurement device 10AL and the measurement device 10AR as a distance measurement sensor, the position of each part of the user U (e.g., left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, etc.) can be measured by using the depth data obtained by the sensor 111.

[0521] 3.3.5 Method for Measuring the Shape of the Hand / Fingers

[0522] The shape of the fingers of the user U can be similar to that in the above-mentioned first embodiment Figures 13 to 16 as described in the measurement method.

[0523] 3.4 Coordinate System

[0524] Next, the integration of the coordinate systems of the reference measurement device 310H and the plurality of measurement devices 10 will be described. Figure 91 is a schematic diagram for showing the integration of the coordinate system according to the present embodiment.

[0525] As described above, there is known information about the positions of the plurality of sensors 111 included in the reference measurement device 310H in the client coordinate system or the server map coordinate system of the reference measurement device 310H. In the client coordinate system or the server map coordinate system of the reference measurement device 310H, the positions of the measurement devices 10 attached to the two wrists of the user U can be measured by using the image data obtained by at least two sensors 111 of the reference measurement device 310H by triangulation.

[0526] On the other hand, the positions of the parts of the user U to which the measurement devices 10 are not attached in the client coordinate system or the server map coordinate system of the reference measurement device 310H can be measured by using the image data obtained by the measurement devices 10 attached to the two wrists by triangulation.

[0527] Subsequently, the position of the reference measurement device 310H attached to the head (or torso) of the user U in the client coordinate system or the server map coordinate system of the reference measurement device 310H can be estimated by using SLAM or positioning of the IMU 112 and / or the GNSS receiver 311.

[0528] From the above, by using a method similar to the method described with reference to Figure 18 etc., the client map coordinate systems of each of the reference measurement device 310H and the plurality of measurement devices 10 can be integrated into a common server map coordinate system.

[0529] 3.5 Combinatorial deformation of units performing respective processes

[0530] Here, for rearrangement, a list of processes for implementing the above functions is shown in Figure 92 、 Figure 95 and Figure 98 ; a list of output data for each process is shown in Figure 93 、 Figure 96 and Figure 99 ; and a list of combinatorial variations of units performing each process is shown in Figure 94 、 Figure 97 and Figure 100 Note that Figures 92 to 94 shows the case where the position of the wrist is measured and reflected on a human model such as an avatar, Figures 95 to 97 shows the case where the positions of the elbow and shoulder are measured and reflected on a human model such as an avatar, Figures 98 to 100 shows the case where the positions of the knee, ankle, and torso are measured and reflected on a human model such as an avatar.

[0531] As shown in Figures 92 to 100 , the process from the process of measuring the position of each unit to the process of reflecting and displaying the measured position on the human model can be divided into multiple processes, and each separated process can be executed in any of the reference measurement device 310H, the measurement device 10, the relay device 20, or the information processing device 40. Note that Figures 92 to 100 The processes, output data, and combinatorial variation examples shown in

[0532] 3.6 Overview

[0533] As described above, according to the present embodiment and its modifications, the reference measurement device 310H attached to the head or torso of the user U can be used as a reference to measure the self-positions of the measurement devices 10AL and 10AR. Therefore, similar to the first embodiment and its variations described above, even when the positions of the measurement devices 10AL and 10AR move freely, the positions of each part of the user U can be measured. There is no need to use SLAM in the measurement devices 10AL and 10AR, so the possibility of reducing the processing amount of the entire system is increased.

[0534] 4. Application Embodiment

[0535] Next, applications of the human body tracking system using the above-described embodiment or its variations will be described through some embodiments.

[0536] 4.1 First Application Embodiment

[0537] As advantages obtained from the configuration and operation of the system, the human body tracking system according to the above-described embodiment may have the following exemplary effects in addition to the above effects.

[0538] · Improve the accuracy and reproducibility of 3D reconstruction by using techniques such as Structure from Motion (SfM).

[0539] · Since the sensors are attached to the human body, there are no blind spots, which is a difference from fixed sensors.

[0540] · Capable of detecting detailed movements of expressions and fingertips, as well as detailed surrounding situations.

[0541] · Capable of achieving more refined 3D reconstruction and display at locations with a high population density in the real world.

[0542] · Enable the exchange between people in the virtual space and people in the real space.

[0543] · The usability in various scenarios is not limited to meetings, such as sports, training, and events (live events, guided tours, performances, dates, etc.).

[0544] · The reduction in the number of sensors results in a reduction in installation time and effort.

[0545] Considering these effects, the human body tracking system according to the above-described embodiment or its modifications will likely be particularly effective for applications such as volume capture or fused XR free viewpoint technology that require generating 3D data by installing large-scale facilities including a large number of cameras and performing imaging in places such as dedicated studios and stadiums.

[0546] In addition, due to blind spots created by obstructions such as people or objects in past system configurations, the human body tracking system according to the above-described embodiment or its modification is particularly effective for applications that are probably difficult in terms of precise 3D representation of details.

[0547] On the other hand, in past gatherings such as online meetings using virtual spaces, there have been cases where it is difficult and uncomfortable for both real-space participants and virtual-space participants to exist in the same space, including extreme usage patterns where all participants are in the real space or all participants are in the virtual space.

[0548] In addition, in order for both parties to communicate with each other in a gathering such as an online meeting, the surrounding environment or a person in one real space needs to be expressed in 3D towards a person in another real space. However, there is a problem that large-scale facilities must be provided separately in two real spaces. In addition, even in such a case, due to blind spots caused by obstructions such as people or objects, there are also problems in performing precise 3D representation of details.

[0549] In this case, by introducing the human body tracking system according to the present disclosure, the number of fixed cameras for imaging the external environment and the like can be reduced, and the system configuration can be reduced, thereby achieving a significant reduction in the introduction cost. In addition, since sensors attached to people are used to track the actions and postures of people, it is also possible to suppress a decrease in the reproduction accuracy due to blind spots and the like.

[0550] Figure 101 is a schematic diagram showing a schematic configuration example of an online meeting system according to a first application embodiment. In this application embodiment, for simplicity, the case where the human body tracking system 1B according to the second modification of the first embodiment described above is introduced into the online meeting system 400 will be described as an example. Figure 51 The case where the human body tracking system 1B according to the second modification of the first embodiment described above is introduced into the online meeting system 400.

[0551] As Figure 101 shown, the online meeting system 400 according to the first application embodiment has such a configuration that points ST1 and point ST2 located away from point ST1 are connected to the server 470 via the network 60.

[0552] Point ST1 has the human body tracking system 1B and a plurality of cameras 411A introduced therein. Various information such as image data (which may be video data) captured by the measuring device 10 of the human body tracking system 1B and each camera 411A and the position information of the user U1 is transmitted to the server 470 via the network 60. For example, the human body tracking system 1B should be worn by the user U1 present at point ST1. In addition, the plurality of cameras 411A can be fixed at positions where the entire state of point ST1 can be imaged.

[0553] On the other hand, similar to the point ST1, the point ST2 has been introduced into the human body tracking system 1B and the plurality of cameras 411B. Various information such as image data (which may be video data) captured by the measuring device 10 of the human body tracking system 1B and each camera 411B and the position information of the user U2 are transmitted to the server 470 via the network 60. For example, the human body tracking system 1B should be worn by the user U2 present at the point ST2. In addition, the plurality of cameras 411B may be fixed at positions where the entire state of the point ST2 can be imaged.

[0554] The point ST2 may be a space designed similar to the point ST1. For example, in terms of the space and layout dimensions of the installed furniture and items, the point ST2 may be designed to be similar to the point ST1.

[0555] In addition, the online conferencing system 400 may have the user U3 present as a participant at a position different from the points ST1 and ST2. The user U3 wears the human body tracking system 1B. Various information such as image data (which may be video data) captured by the measuring device 10 of the human body tracking system 1B and the position information of the user U3 are transmitted to the server 470 via the network 60.

[0556] The server 470 integrates the coordinate systems of the point ST1 and the point ST2 with each other, displays the avatar C1 of the user U1 present at the point ST1 on the HMD 80 of the user U2 present at the point ST2, and controls the avatar C1 according to the actions of the user U1. Similarly, the server 470 displays the avatar C2 of the user U2 present at the point ST2 on the HMD 80 of the user U1 present at the point ST1, and controls the avatar C2 according to the actions of the user U2. Incidentally, the HMDs 80 attached to the users U1 and U2 present at the points ST1 and ST2 respectively may both be optical see-through HMDs or video see-through HMDs.

[0557] In addition, the space at the point ST1 or the point ST2 reproduced from the image data captured by the cameras 411A and / or 411B is displayed as a virtual space on the HMD 80 of the user U3 who is a VR participant. In addition, the avatar C1 of the user U1 present at the point ST1 and the avatar C2 of the user U2 present at the point ST2 are also displayed in the virtual space, and these avatars are controlled according to the actions of the users U1 and U2 respectively.

[0558] In addition, the server 470 also integrates the coordinate system of the body tracking system 1B worn by the user U3 who is a VR participant onto the coordinate systems of the points ST1 and ST2. Further, the server 470 also displays the avatar C3 of the user U3 who is a VR participant on the HMDS 80 of the users U1 and U2 existing at the points ST1 and ST2, respectively, and controls the avatar C3 according to the actions of the user U3.

[0559] In addition, an object OB1 targeted by any one of the users can be displayed as a virtual object Cob1 on the HMDS 80 of the users U1 to U3.

[0560] For example, the online conferencing system 400 can be introduced not only in an ordinary office meeting room but also in various online facilities that can connect two or more remote bases, such as rental offices, cafes, karaoke shops, event sites (including concert sites and performing arts stages), fitness facilities, and golf driving ranges.

[0561] 4.2 Second Application Embodiment

[0562] Next, a case where the body tracking system according to the present disclosure is applied to an application for observing the body activity postures of users will be described.

[0563] Conventionally used applications for observing the body activity postures of users have used techniques for measuring the observation positions during the body activities of users by attaching IMUs, bending sensors, etc. to all joints to be measured by the users, or techniques for measuring the observation positions in the body activities of users by installing multiple cameras indoors and based on the video images captured by the cameras.

[0564] However, attaching sensors such as IMUs to all measurement target points of the users causes discomfort to the users when wearing and also hinders natural movement. On the other hand, the technique of using cameras installed indoors to measure users is restricted by the location, making it difficult to measure the body activity postures of users in places such as outdoors.

[0565] In these cases, by introducing the body tracking system according to the present disclosure, the actions and postures of the entire body of the user can be measured using a small number of sensors, making it possible to suppress drawbacks such as discomfort when wearing, interference with natural movement, and restrictions on location.

[0566] Figure 102 It is a flowchart showing a schematic operation embodiment of a body activity posture observation system according to the second application embodiment. For clarity, this description will describe an exemplary case of introducing the body tracking system 1.

[0567] As Figure 102As shown, in this operation, first, using the technology according to the above-described embodiments or their modified examples, the client map coordinate system of each measurement device 10 is integrated into the server map coordinate system (step S411).

[0568] Next, for example, the appearance of the user is imaged (step S412). The image data obtained by imaging can be input to the information processing device 40. For example, imaging of the user can be performed by using a device such as a smart phone including a distance measurement sensor.

[0569] Next, a 3D model (which may be a skeleton model) of the external shape of the user is created based on the image data obtained by imaging the user (step S413). At this time, the 3D model can be gradually displayed on the display device 50 to present the progress of the creation of the 3D model to the user. In response thereto, the user can additionally input the image data obtained by re-imaging the missing part to the information processing device 40, and the information processing device 40 can update the 3D model based on the additionally input image data.

[0570] When the 3D model of the user is completed as described above, measurement of the body activity posture of the user using the human body tracking system 1 is performed (step S414). For example, the position information of each part of the user estimated by the measurement (hereinafter, also referred to as body activity posture information) can be accumulated in the relay device 20. Not limited thereto, this information can be stored in any measurement device 10 or information processing device 40. In addition, the body activity posture information may include the position information of the entire user (i.e., information about the movement of the user as a whole person).

[0571] Note that the body activity of the user to be set as the measurement target can be outdoor body activity or indoor body activity. In addition, the body activity can be body activity without using an instrument (e.g., running or manual body activity), body activity using an instrument that does not involve position movement of the instrument (e.g., a treadmill), or body activity using an instrument that involves position movement of the instrument (e.g., a golf swing or a tennis swing). In the case of body activity using an instrument that involves position movement of the instrument, a technique similar to the technique of estimating the position of each part of the user by using two measurement devices 10 attached to the two wrists of the user can be used to estimate the position and posture of the instrument. In addition, in order to facilitate the detection of the position of the instrument by the measurement device 10, a transmitting device that emits a signal (e.g., a beacon) can be attached to the instrument.

[0572] Thereafter, when the measurement of the user's body activity posture is completed, the body activity posture information stored in the relay device 20 is input to the information processing device 40 (step S415). It should be noted that the body activity posture information can be directly input from the measurement device 10 to the information processing device 40, or can be input via the relay device 20.

[0573] Next, the information processing device 40 analyzes the body activity posture (step S416). Subsequently, the information processing device 40 controls the 3D model of the user based on the analyzed body activity posture (step S417). Subsequently, the video obtained by rendering the 3D model of the user moving according to the body activity posture is displayed on the display device 50 (step S418), and then this operation ends. Note that the 3D model displayed here can reflect the appearance of the user (clothes, hat, accessories, etc.) imaged when creating the 3D model.

[0574] For example, in the case of body activity in an environment equipped with the display device 50, it is allowed to transmit the measurement result to the information processing device 40 in real time during the body activity, and the action of the 3D model based on the measurement result can be displayed on the display device 50 in real time, rather than accumulating the measurement result in the relay device 20 during the body activity and displaying the result after the body activity.

[0575] Alternatively, for example, in the case of observing the body activity posture during a specific period (such as a golf swing practice), the user's body activity can be measured and accumulated during the specific period. When a predetermined trigger occurs, the measurement result can be sent to the information processing device 40, and the 3D model reflecting the measurement result can be displayed on the display device 50.

[0576] 4.3 Third Application Embodiment

[0577] Next, the case where the human body tracking system according to the present invention is applied to action tracking using key point recognition will be described.

[0578] The embodiments of applying the human body tracking system according to the present invention to action tracking using key point recognition include: an embodiment of using the human body tracking system to restore the entire body of the user in 3D (the first embodiment); and an example of using the human body tracking system to restore the feature points of the user (for example, parts such as joints) (the second embodiment).

[0579] (The First Embodiment)

[0580] Figure 103 is a flowchart showing the processing flow according to the first embodiment. Note that the integration of the coordinate system in the measurement device 10 should be completed before the Figure 103 operation shown.

[0581] AsFigure 103 As shown, the first step of the first embodiment is to input the image data acquired by each measuring device 10 into the information processing device 40 (step S421).

[0582] In response thereto, the information processing device 40 performs three-dimensional image processing on each input image data (step S422). This operation creates three-dimensional point cloud data of the user's entire body (step S423). Examples of the three-dimensional image processing in step S422 may include SfM and stereo vision.

[0583] Next, the information processing device 40 analyzes the created three-dimensional point cloud data to identify key points in the three-dimensional point cloud data (step S424).

[0584] Next, the information processing device 40 specifies the movement of the identified key points to identify the user's posture and actions (step S425). Subsequently, the information processing device 40 reflects the identified posture and actions in the user's 3D model to move the 3D model according to the user's actions (step S426).

[0585] Then, a determination is made as to whether to end the current operation (step S427). In the case where it is determined to end the current operation (yes in step S427), the current operation ends. On the other hand, in the case where the current action has not ended (step S427: no), the process returns to step S421 for subsequent actions.

[0586] (Second Embodiment)

[0587] Figure 104 is a flowchart for showing the processing flow according to the second embodiment. Note that the integration of the coordinate systems in the measuring device 10 should be completed before the Figure 104 operation shown.

[0588] As Figure 104 shown, similar to the first embodiment, the first step of the second embodiment is to input the image data acquired by each measuring device 10 into the information processing device 40 (step S431).

[0589] In response thereto, the information processing device 40 analyzes each input image data to identify key points of the joints in the user (step S432).

[0590] Next, the information processing device 40 performs three-dimensional image processing on the image data of the joints including the identified key points (step S433). This operation creates three-dimensional point cloud data of the user's joint parts (step S434). Examples of the three-dimensional image processing in step S433 may include SfM and stereo vision.

[0591] Next, the information processing device 40 specifies the movement of the recognized joint to recognize the user's posture and motion (step S435). Subsequently, the information processing device 40 reflects the recognized posture and motion in the user's 3D model to move the 3D model according to the user's motion (step S436).

[0592] After that, it is determined whether to end the current operation (step S437), and when it is determined to end (Yes in step S437), the current operation ends. On the other hand, when this operation has not ended (step S437: No), the process returns to step S431 to execute the subsequent operation.

[0593] 5. Hardware Configuration

[0594] For example, the measurement device 10, the reference measurement device 310H, the relay device 20, the information processing device 40, the servers 70 and 470, and the HMD 80 according to the above-described embodiments, their modified examples, and application examples can be implemented by a computer 1000 having the configuration shown in Figure 105 . Figure 105 FIG. is a hardware configuration diagram showing an embodiment of the computer 1000, and the computer 1000 implements the functions of the measurement device 10, the reference measurement device 310H, the relay device 20, the information processing device 40, the servers 70 and 470, and the HMD 80. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each component of the computer 1000 is interconnected by a bus 1050.

[0595] The CPU 1100 operates based on programs stored in the ROM 1300 or the HDD 1400 to control each component. For example, the CPU 1100 expands the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0596] The ROM 1300 stores a boot program such as a basic input / output system (BIOS) executed by the CPU 1100 when the computer 1000 starts up, programs dependent on the hardware of the computer 1000, and the like.

[0597] The HDD 1400 is a non-transitory computer-readable recording medium that records programs executed by the CPU 1100, data used by the programs, and the like. Specifically, the HDD 1400 is a recording medium that records programs for performing various operations according to the present disclosure, which is an example of program data 1450.

[0598] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices via the communication interface 1500 or transmits data generated by the CPU 1100 to other devices.

[0599] The input / output interface 1600 has a configuration including the above-described I / F unit 18 and is an interface for connecting the input / output device 1650 and the computer 1000 to each other. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. In addition, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. Further, the input / output interface 1600 can be used as a medium interface for reading programs and the like recorded on a predetermined recording medium (or media). Examples of the medium include optical recording media such as a digital versatile disc (DVD) or a phase change rewritable disc (PD), magneto-optical recording media such as a magneto-optical disc (MO), tape media, magnetic recording media, and semiconductor memories.

[0600] For example, when the computer 1000 is used as the measuring device 10, the reference measuring device 310H, the relay device 20, the information processing device 40, the server 70 and 470, and the HMD 80 according to the above-described embodiments, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to implement the functions of the measuring device 10, the reference measuring device 310H, the relay device 20, the information processing device 40, the server 70 and 470, and the HMD 80. In addition, the HDD 1400 stores programs and the like according to the present disclosure. As another example, when the CPU 1100 executes program data 1450 read from the HDD 1400, the CPU 1100 can obtain these programs from another device via the external network 1550.

[0601] 6. Conclusion

[0602] For example, the above technology is specified as follows. One of the disclosed technologies is the information processing device 40. As described in reference Figures 1 to 100 etc., the information processing device 40 includes a processing unit 141 configured to measure the positions of a plurality of parts (body parts) of a user U based on the positions (self-positions) of a plurality of measuring devices 10 (e.g., measuring devices 10AL and 10AR) attached to the user U to acquire image data of the user U and based on the image data acquired by each of the plurality of measuring devices 10. The respective positions of the plurality of measuring devices 10 are self-estimated or measured using another measuring device (e.g., the reference measuring device 310H) attached to the user U.

[0603] Based on the above information processing device 40, the positions of multiple parts of the user U are measured using the estimated or measured positions of the measurement device 10. Therefore, even if the measurement device 10 moves freely in position, the positions of the parts of the user U can be measured.

[0604] As described with reference to the figure such as Figures 1 to 55 As described in the figure, each of the measurement devices 10 can be attached to any one of multiple parts of the user U. The processing unit 141 can estimate the position of the place equipped with these measurement devices 10 by SLAM using the image data separately acquired by these measurement devices 10. The processing unit 141 can measure the position of at least one part of the multiple parts by triangulating the image data acquired by at least two of the multiple measurement devices 10 (for example, the measurement devices 10AL and 10AR). The processing unit 141 can measure the positions of the parts (such as the elbows and shoulders) between the reference part (such as the torso) of the multiple parts and the part whose position has been estimated (such as the wrist) based on the skeleton model of the human body. For example, the position of the measurement device 10 can be estimated, or the position of each unit of the user U can be measured in this way.

[0605] As described with reference to Figures 56 to 77 As described in etc., each measurement device 10 (for example, the measurement device 10AL and the measurement device 10AR) can be attached to any one of multiple parts of the user U. The processing unit 141 can measure the position of the part equipped with each measurement device 10 using a marker (such as bip MZ) attached to the user U as a reference. For example, the position of the measurement device 10 can be estimated, or the position of each part of the user U can be measured in this way.

[0606] The measurement device 10B can be attached to the user U to suppress changes in the positional relationship with the marker MZ, and the processing unit 141 can measure the position of the field point equipped with the measurement device 10B by SLAM using the image data acquired by the measurement device 10B. Even if the position of the marker MZ changes, the positions of the measurement devices 10AL and 10AR can be estimated with reference to the position of the marker MZ, and the positions of each part of the user U can be measured.

[0607] As referred to Figures 78 to 100As described in, for example, [references], the measurement devices 10 (e.g., measurement devices 10AL and 10AR) can each be attached to any one of a plurality of parts of the user U. The processing unit 141 can measure the position of the part of the user U equipped with the measurement device 10 among the plurality of parts by using triangulation based on the image data of each measurement device 10 acquired by the reference measurement device 310H including at least two sensors 111. The processing unit 141 can estimate the position of the reference measurement device 310H by using the image data acquired by the reference measurement device 310H through SLAM. For example, the position of the measurement device 10 can be estimated, or the position of each part of the user U can be measured in this way.

[0608] As seen in Figure 18 , Figures 22 to 34 , Figures 57 to 61 , Figure 91 As described in, for example, [references], the processing unit 141 can integrate the first coordinate system (client map coordinate system) set in each measurement device 10 (e.g., measurement devices 10AL and 10AR) into a common second coordinate system (server map coordinate system) in order to measure the positions of the plurality of parts in the second coordinate system. The processing unit 141 can integrate the first coordinate system of the individual measurement devices 10 based on the image data, which is obtained by the measurement device 10 individually imaging a marker (e.g., marker MK) provided in the external environment or any one of the imaging devices in the plurality of measurement devices 10. For example, the positions of the plurality of parts can be measured by integrating the coordinate systems in this way.

[0609] As described in reference to the drawings such as Figure 2 , the measurement device 10 (e.g., measurement devices 10AL and 10AR) can each include at least one of an image sensor, a distance measurement sensor, and an EVS as the sensor 111 for acquiring image data. For example, the position of the body part of the user U can be measured based on the image data acquired by the measurement device 10 including such sensors.

[0610] The plurality of measurement devices 10 can be two measurement devices 10 (measurement devices 10AL and 10AR). By using the minimum number of measurement devices 10 to measure the positions of a large number of parts of the user U, the discomfort during wearing or the risk of hindrance to movement in the user U can be minimized.

[0611] The parts of the user U can include at least one of the left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, left ankle, right ankle, left knee, right knee, and head. For example, the positions of these different parts of the user U can be estimated.

[0612] The processing unit 141 can control the position (posture, movement, etc.) of an avatar (or character) in the virtual space corresponding to the user U based on the estimated positions of multiple parts of the user U.

[0613] The information processing method described with reference to the drawings such as Figures 1 to 100 is also one of the disclosed technologies. The information processing method includes measuring the positions of multiple parts of the user U based on the positions of multiple measurement devices 10 (for example, measurement devices 10AL and 10AR) attached to the user U to obtain image data of the user U and based on the image data obtained by each of the multiple measurement devices 10. The respective positions of the multiple measurement devices 10 are self-estimated or measured using another measurement device (for example, the reference measurement device 310H) attached to the user U. Even with such an information processing method, as described above, even when the positions of the measurement devices 10 move freely, the positions of the body parts of the user U can be measured.

[0614] As described with reference to the drawings (such as Figures 1 to 55 ), each measurement device 10 can be attached to any one of multiple parts of the user U, and the position of the part equipped with each measurement device 10 can be estimated by SLAM using the image data obtained by each measurement device 10. The position of at least one of the multiple parts can be measured by using triangulation based on the image data obtained by at least two of the multiple measurement devices 10 (for example, measurement devices 10AL and 10AR). It is allowed to measure the positions of parts (such as elbows and shoulders) between a reference part (such as the torso) and a part whose position has been estimated (such as the wrist) in the multiple parts based on the skeletal model of the human body. For example, the position of the measurement device 10 can be estimated, or the position of each unit of the user U can be measured in this way.

[0615] As described with reference to the drawings such as Figures 56 to 77 , each measurement device 10 (measurement device 10AL and measurement device 10AR) can be attached to any one of multiple parts of the user U. The position of the part equipped with each measurement device 10 can be measured with a marker (for example, bib MZ) attached to the user U as a reference. For example, the position of the measurement device 10 can be estimated, or the position of each part of the user U can be measured in this way.

[0616] As described with reference to Figures 78 to 100As described in, for example, [references], the measurement devices 10 (e.g., measurement devices 10AL and 10AR) can each be attached to any one of a plurality of parts of the user U. The position of the part of the plurality of parts equipped with the measurement device 10 can be measured by using triangulation based on the image data of each measurement device 10, and the image data of each measurement device 10 is acquired by a reference measurement device 310H including at least two sensors 111. The position of the reference measurement device 310H can be estimated by SLAM using the image data acquired by the reference measurement device 310H. For example, the position of the measurement device 10 can be estimated, or the position of each part of the user U can be measured in this way.

[0617] As described in Figure 18 , Figures 22 to 34 , Figures 57 to 61 , Figure 91 As described in, for example, [references], the information processing method can further include integrating the first coordinate system (client map coordinate system) set in each measurement device 10 (e.g., measurement devices 10AL and 10AR) into a common second coordinate system (server map coordinate system), and the positions of the plurality of parts can be measured as positions in the second coordinate system. The first coordinate system of each measurement device 10 can also be configured to be synthesized into the second coordinate system based on the image data obtained by imaging, by each measurement device 10, a marker (e.g., marker MK) provided in the external environment or any one of the plurality of measurement devices 10. For example, the positions of the plurality of parts can be measured by integrating the coordinate systems in this way.

[0618] As described with reference to the drawings such as Figure 2 , the measurement device 10 (e.g., measurement devices 10AL and 10AR) can each include at least one of an image sensor, a distance measurement sensor, and an EVS as the sensor 111 for acquiring image data. For example, the position of the body part of the user U can be measured based on the image data acquired by the measurement device 10 including such sensors.

[0619] The embodiments of the present disclosure have been described above. However, the technical scope of the present disclosure is not limited to the above embodiments, and various modifications can be made without departing from the scope of the present disclosure. In addition, components in different embodiments and variations can be appropriately combined.

[0620] The effects described in the respective embodiments of this specification are only examples, and thus, there may be other effects, not limited to the exemplary effects.

[0621] In addition, the above-described embodiments can be used alone or in combination with other embodiments.

[0622] It should be noted that the present technology can also have the following configuration.

[0623] (1) An information processing device, comprising:

[0624] A processing unit configured to measure the positions of a plurality of parts of a user based on the positions of a plurality of measurement devices respectively attached to the user, so as to obtain image data of the user and image data obtained by each of the plurality of measurement devices, wherein

[0625] The position of each of the plurality of measurement devices is self-estimated by the measurement device or measured using another measurement device attached to the user.

[0626] (2) The information processing device according to (1), wherein

[0627] Each measurement device is attached to any one of a plurality of parts of the user, and

[0628] The processing unit estimates the positions of the parts equipped with each of these measurement devices by using Simultaneous Localization and Mapping (SLAM) of the image data obtained by each of these measurement devices.

[0629] (3) The information processing device according to (1) or (2), wherein

[0630] The processing unit measures the position of at least one of the plurality of parts by using triangulation based on the image data obtained by at least two of the plurality of measurement devices.

[0631] (4) The information processing device according to any one of (1) to (3), wherein

[0632] The processing unit measures the position of the part between the reference part and the part with the estimated position among the plurality of parts based on the skeleton model of the human body.

[0633] (5) The information processing device according to (1), wherein

[0634] Each measurement device is attached to any one of a plurality of parts of the user, and

[0635] The processing unit uses the marker attached to the user as a reference to measure the position of the part equipped with each measurement device.

[0636] (6) The information processing device according to (1), wherein

[0637] Each measurement device is attached to any one of a plurality of parts of the user, and

[0638] The processing unit measures the positions of the parts equipped with the measuring devices among the multiple parts by using triangulation based on the image data of each measuring device, and the image data is acquired by a reference measuring device including at least two sensors.

[0639] (7) The information processing device according to (6), wherein

[0640] The processing unit estimates the position of the reference measuring device by using the image data collected by the reference measuring device through SLAM.

[0641] (8) The information processing device according to any one of (1) to (7), wherein

[0642] The processing unit integrates the first coordinate system already set in each measuring device into a common second coordinate system to measure the positions of the multiple parts in the second coordinate system.

[0643] (9) The information processing device according to (8), wherein

[0644] The processing unit integrates the first coordinate system of the individual measuring device into the second coordinate system based on the image data obtained by imaging a marker set in the external environment or any one of the multiple measuring devices by the measuring device alone.

[0645] (10) The information processing device according to any one of (1) to (9), wherein

[0646] Each measuring device includes at least one of an image sensor, a distance measuring sensor, or an EVS as the sensor for acquiring image data.

[0647] (11) An information processing method, comprising:

[0648] Measuring the positions of multiple parts of a user based on the positions of multiple measuring devices respectively attached to the user to obtain the image data of the user and the image data acquired by each of the multiple measuring devices, wherein

[0649] The position of each of the multiple measuring devices is self-estimated by the measuring device or measured by using another measuring device attached to the user.

[0650] (12) The information processing method according to (11), wherein,

[0651] Each measuring device is attached to any one of the multiple parts of the user, and

[0652] The positions of the parts equipped with each of these measuring devices are estimated by using Simultaneous Localization and Mapping (SLAM) of the image data acquired by each of these measuring devices.

[0653] (13) The information processing method according to (11) or (12), wherein

[0654] Based on the image data obtained by at least two of the plurality of measurement devices, the position of at least one of the plurality of parts is measured by using triangulation.

[0655] (14) The information processing method according to any one of (11) to (13), wherein

[0656] Based on the skeleton model of the human body, the position of the part between the reference part and the part with the estimated position among the plurality of parts is measured.

[0657] (15) The information processing method according to (11), wherein

[0658] Each measurement device is attached to any one of the plurality of parts of the user, and

[0659] The position of the part equipped with each measurement device is measured by using the marker attached to the user as a reference.

[0660] (16) The information processing method according to (11), wherein

[0661] Each measurement device is attached to any one of the plurality of parts of the user, and

[0662] The position of the part equipped with the measurement device among the plurality of parts is measured by using triangulation based on the image data of each measurement device, and the image data is obtained by a reference measurement device including at least two sensors.

[0663] (17) The information processing method according to (16), wherein

[0664] SLAM uses the image data collected by the reference measurement device to estimate the position of the reference measurement device.

[0665] (18) The information processing method according to any one of (11) to (17) further includes:

[0666] Integrating the first coordinate system already set in each measurement device into a common second coordinate system, wherein

[0667] The positions of the plurality of parts are measured as positions in the second coordinate system.

[0668] (19) The information processing method according to (18), wherein

[0669] Integrate the first coordinate system of the individual measuring device into the second coordinate system based on image data obtained by imaging, by the individual measuring device alone, a marker arranged in an external environment or any one of a plurality of measuring devices.

[0670] (20) The information processing method according to any one of (11) to (19), wherein

[0671] each measuring device includes at least one of an image sensor, a distance measuring sensor, or an EVS as a sensor for acquiring the image data.

[0672] List of reference numerals

[0673] 1, 1A to 1E, 2, 2A, 2B, 3, 3A Human tracking system

[0674] 10, 10AL, 10AR, 10B, 10FL, 10FR, 10G, 10H Measuring device

[0675] 20, 220 Relay device

[0676] 30 Communication device

[0677] 40 Information processing device

[0678] 50 Display device

[0679] 60 Network

[0680] 70, 470 Server

[0681] 80 HMD

[0682] 101-1, 101-2 Housing

[0683] 102-1, 102-2 Lens

[0684] 103 Band

[0685] 111, 111-1 to 111-N Sensor

[0686] 111g Non-optical sensor

[0687] 112, 221 IUM

[0688] 113, 121, 141 Processing unit

[0689] 114, 122, 142 Recording unit

[0690] 115, 123, 143 Communication unit

[0691] 116, 124, 145 Power supply

[0692] 144 Image Processing Unit

[0693] 222, 311 GNSS Receiver

[0694] 310H, 310HD Reference Measurement Device

[0695] 411A, 411B Cameras.

Claims

1. An information processing device, comprising: a processing unit configured to measure positions of a plurality of parts of a user based on positions of a plurality of measurement devices to obtain image data of the user and image data obtained by each of the plurality of measurement devices, each of the plurality of measurement devices being attached to the user, wherein, the position of each of the plurality of measurement devices is self-estimated by the measurement device or measured using another measurement device attached to the user.

2. The information processing device according to claim 1, wherein, each of the measurement devices is attached to any one of the plurality of parts of the user, and the processing unit estimates the position of the part equipped with each of the plurality of measurement devices by using Simultaneous Localization and Mapping (SLAM) with the image data obtained by each of the plurality of measurement devices.

3. The information processing device according to claim 1, wherein, the processing unit measures the position of at least one of the plurality of parts by using triangulation based on the image data obtained by at least two of the plurality of measurement devices.

4. The information processing device according to claim 1, wherein, the processing unit measures the position of a part between a reference part and a part whose position has been estimated among the plurality of parts based on a skeleton model of a human body.

5. The information processing device according to claim 1, wherein, each of the measurement devices is attached to any one of the plurality of parts of the user, and the processing unit measures the position of the part equipped with each measurement device using a marker attached to the user as a reference.

6. The information processing device according to claim 1, wherein, each of the measurement devices is attached to any one of the plurality of parts of the user, and the processing unit measures the position of the part equipped with the measurement device among the plurality of parts by using triangulation based on the image data of each measurement device, the image data being obtained by a reference measurement device including at least two sensors.

7. The information processing device according to claim 6, wherein, the processing unit estimates the position of the reference measurement device by using SLAM with the image data obtained by the reference measurement device.

8. The information processing device according to claim 1, wherein, the processing unit integrates a first coordinate system already set in each of the measurement devices into a common second coordinate system to measure the positions of the plurality of parts in the second coordinate system.

9. The information processing device according to claim 8, wherein, the processing unit integrates the first coordinate system of the individual measurement devices into the second coordinate system based on image data obtained by individually imaging a marker set in an external environment or in any one of the plurality of measurement devices by the plurality of measurement devices.

10. The information processing device according to claim 1, wherein, Each of the measurement devices includes at least one of an image sensor, a distance measurement sensor, and an EVS as a sensor for acquiring the image data.

11. An information processing method, comprising: Measuring the positions of a plurality of parts of a user based on the positions of a plurality of measurement devices to obtain the image data of the user and the image data obtained by each of the plurality of measurement devices, wherein each of the plurality of measurement devices is attached to the user, and The position of each of the plurality of measurement devices is self-estimated by the measurement device or measured using another measurement device attached to the user.

12. The information processing method according to claim 11, wherein Each of the measurement devices is attached to any one of the plurality of parts of the user, and Using Simultaneous Localization and Mapping (SLAM) with the image data obtained by each of the plurality of measurement devices to estimate the position of the part equipped with each of the plurality of measurement devices.

13. The information processing method according to claim 11, wherein Measuring the position of at least one of the plurality of parts based on triangulation using the image data obtained by at least two of the plurality of measurement devices.

14. The information processing method according to claim 11, wherein Measuring the position of a part between a reference part and a part whose position has been estimated among the plurality of parts based on a skeleton model of the human body.

15. The information processing method according to claim 11, wherein Each of the measurement devices is attached to any one of the plurality of parts of the user, and Using a marker attached to the user as a reference to measure the position of the part equipped with each measurement device.

16. The information processing method according to claim 11, wherein Each of the measurement devices is attached to any one of the plurality of parts of the user, and Measuring the position of the part equipped with the measurement device among the plurality of parts based on triangulation using the image data of each measurement device, the image data being obtained by a reference measurement device including at least two sensors.

17. The information processing method according to claim 16, wherein Estimating the position of the reference measurement device using the image data obtained by the reference measurement device by SLAM.

18. The information processing method according to claim 11, further comprising: Integrating a first coordinate system already set in each of the measurement devices into a common second coordinate system, wherein The positions of the plurality of parts are measured as positions in the second coordinate system.

19. The information processing method according to claim 18, wherein Integrating the first coordinate system of an individual measurement device into the second coordinate system based on image data obtained by individually imaging a marker provided in an external environment or in any one of the plurality of measurement devices by the plurality of measurement devices.

20. The information processing method according to claim 11, wherein each of the measurement devices includes at least one of an image sensor, a distance measurement sensor, and an EVS as a sensor for acquiring the image data.

Images