Information processing system, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing method and an information processing program capable of measuring a distance on a real space with a simpler operation.SOLUTION: An information processing device includes an acquisition part, an estimation part, a measurement part, and an output control part. The acquisition part acquires an image captured by an imaging part, and position information of the imaging part. The estimation part estimates a first position on a real space on the basis of a first image captured by the imaging part at a first view point before moving the imaging part by a user, and position information of the first view point, and estimates a second position on the real space on the basis of a second image captured by the imaging part at a second view point after movement of the imaging part, position information of the second view point and an object recognition result. The measurement part measures a distance between the first position and the second position. The output control part performs control of displaying display information including a measurement result of the measurement part.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] 2. Description of the Related Art In recent years, with the advancement of image recognition technology, it has become possible to recognize the position and orientation of a real object (that is, an object in real space) contained in an image captured by an imaging device. As one of the applications of such object recognition, a technology called Augmented Reality (AR) is known. By using the AR technology, it becomes possible to superimpose various forms of virtual content (hereinafter also referred to as "virtual objects") such as text, icons, or animations on real objects captured in an image of real space and present them to a user. For example, Patent Document 1 discloses an example of the AR technology.

[0003] Furthermore, by applying object recognition technology, it is possible to perform so-called self-location estimation for recognizing the position in real space of an imaging unit that captures an image of an object (and thus a device equipped with the imaging unit). One example of a technology for realizing such self-location estimation is a technology called SLAM (simultaneous localization and mapping). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-92964 A Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, there is a demand for a mechanism that enables distance measurement in real space with simpler operations by applying technologies such as the above-mentioned AR and SLAM.

[0006] In view of this, the present disclosure proposes an information processing device, an information processing method, and an information processing program that are capable of measuring distances in real space with simpler operations. [Means for solving the problem]

[0007] According to the present disclosure, there is provided an information processing device including: an acquisition unit that acquires an image captured by an imaging unit and positional information of the imaging unit; an estimation unit that estimates a first position in real space based on a first image captured by the imaging unit at a first viewpoint before a user moves the imaging unit and positional information of the first viewpoint, and estimates a second position in the real space based on a second image captured by the imaging unit at a second viewpoint after the imaging unit is moved, positional information of the second viewpoint, and an object recognition result; a measurement unit that measures the distance between the first position and the second position; and an output control unit that controls the display of display information including the measurement results of the measurement unit.

[0008] Furthermore, according to the present disclosure, there is provided an information processing method including: acquiring an image captured by an imaging unit and position information of the imaging unit; estimating a first position in real space based on a first image captured by the imaging unit at a first viewpoint before a user moves the imaging unit and position information of the first viewpoint; estimating a second position in the real space based on a second image captured by the imaging unit at a second viewpoint after the imaging unit is moved, position information of the second viewpoint, and an object recognition result; measuring a distance between the first position and the second position; and controlling the display of display information including the distance measurement result.

[0009] In addition, according to the present disclosure, an information processing program is provided that causes a computer to execute the following steps: acquire an image captured by an imaging unit and position information of the imaging unit; estimate a first position in real space based on a first image captured by the imaging unit at a first viewpoint before a user moves the imaging unit and position information of the first viewpoint; estimate a second position in the real space based on a second image captured by the imaging unit at a second viewpoint after the imaging unit is moved, position information of the second viewpoint, and an object recognition result; measure a distance between the first position and the second position; and control display of display information including the distance measurement result. Effect of the Invention

[0010] As described above, according to the present disclosure, an information processing device, an information processing method, and an information processing program are provided that are capable of measuring distances in real space with simpler operations.

[0011] It should be noted that the above effects are not necessarily limiting, and any of the effects shown in this specification, or other effects that can be understood from this specification, may be achieved in addition to or instead of the above effects. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram for describing an example of a schematic configuration of an input / output device according to an embodiment of the present disclosure. [Diagram 2] 1 is an explanatory diagram for describing an overview of an information processing system according to a first embodiment of the present disclosure. [Diagram 3] 13A and 13B are diagrams illustrating an example of display information presented in accordance with a distance measurement result. [Figure 4] 2 is a block diagram showing an example of a functional configuration of the information processing system according to the embodiment. FIG. [Diagram 5] 11 is a flowchart showing an example of a flow of a series of processes of the information processing system according to the embodiment. [Figure 6]FIG. 11 is an explanatory diagram for describing an overview of an information processing system according to a second embodiment of the present disclosure. [Figure 7] 11 is a flowchart showing an example of a flow of a series of processes of the information processing system according to the embodiment. [Figure 8] 11 is a flowchart showing an example of a flow of a series of processes of the information processing system according to the embodiment. [Figure 9] 11 is a flowchart showing an example of a flow of a series of processes of the information processing system according to the embodiment. [Figure 10] FIG. 11 is an explanatory diagram for describing an overview of an information processing system according to a third embodiment of the present disclosure. [Figure 11] FIG. 2 is an explanatory diagram for describing an overview of an information processing system according to the embodiment. [Figure 12] 2 is a block diagram showing an example of a functional configuration of the information processing system according to the embodiment. FIG. [Figure 13] 11 is a flowchart showing an example of a flow of a series of processes of the information processing system according to the embodiment. [Figure 14] FIG. 11 is an explanatory diagram for describing an overview of an information processing system according to a fourth embodiment of the present disclosure. [Figure 15] 2 is a block diagram showing an example of a functional configuration of the information processing system according to the embodiment. FIG. [Figure 16] 11 is a flowchart showing an example of a flow of a series of processes of the information processing system according to the embodiment. [Figure 17] FIG. 11 is an explanatory diagram for explaining an overview of an information processing system according to a first modified example. [Figure 18] FIG. 11 is an explanatory diagram for explaining an overview of an information processing system according to Modification 2. [Figure 19] FIG. 11 is an explanatory diagram for explaining an overview of an information processing system according to Modification 3. [Figure 20] 11 is an explanatory diagram for explaining an example of a method for presenting various types of information using the results of measuring distances in real space. FIG. [Figure 21] 11 is an explanatory diagram for explaining an example of a method for presenting various types of information using the results of measuring distances in real space. FIG. [Figure 22] FIG. 13 is a diagram showing an example of a measurement procedure when measuring an area. [Figure 23] FIG. 13 is a diagram showing an example of a measurement procedure when measuring a volume. [Figure 24] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0014] The explanation will be given in the following order. 1. Introduction 1.1. Input / Output Device Configuration 1.2. Principle of self-location estimation 2. First embodiment 2.1.Measurement method 2.2. Functional Configuration 2.3 Processing 2.4.Evaluation 3. Second embodiment 3.1.Measurement method Processing Evaluation 4. Third embodiment 4.1.Measurement method 4.2.Functional Configuration Processing Evaluation 5. Fourth embodiment 5.1.Measurement method 5.2.Functional Configuration Processing Evaluation 6. Variations 6.1. Variation 1: Measuring the length of a curve 6.2. Modification 2: Example of measurement method based on manipulation of virtual objects 6.3. Variation 3: Example of Linking Multiple Devices 7. Application Examples 8. Hardware configuration example 9. Conclusion

[0015] <<1. Introduction>> <1.1. Input / output device configuration> First, an example of a schematic configuration of an input / output device according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram for describing an example of a schematic configuration of an input / output device according to the present embodiment.

[0016] The input / output device 20 according to this embodiment is configured as a so-called head-mounted device that is used by a user by wearing it on at least a part of the head. For example, in the example shown in Fig. 1, the input / output device 20 is configured as a so-called eyewear type (glasses type) device, and at least one of the lenses 293a and 293b is configured as a transmissive display (output unit 211). The input / output device 20 also includes first imaging units 201a and 201b, The input / output device 20 includes second imaging units 203a and 203b, an operation unit 207, and a holder 291 equivalent to a frame of glasses. When the input / output device 20 is worn on the head of a user, the holder 291 holds the output unit 211, the first imaging units 201a and 201b, the second imaging units 203a and 203b, and the operation unit 207 so that they are in a predetermined positional relationship with respect to the head of the user. Although not shown in FIG. 1, the input / output device 20 may include a sound collection unit for collecting the voice of the user.

[0017] Here, a more specific configuration of the input / output device 20 will be described. For example, in the example shown in Fig. 1, the lens 293a corresponds to the lens on the right eye side, and the lens 293b corresponds to the lens on the left eye side. That is, the holder 291 holds the output unit 211 such that the output unit 211 (in other words, the lenses 293a and 293b) are located in front of the user's eyes when the input / output device 20 is worn.

[0018] The first imaging units 201a and 201b are configured as so-called stereo cameras, and are held by the holding unit 291 so as to face the direction in which the user's head faces (i.e., in front of the user) when the input / output device 20 is worn on the user's head. At this time, the first imaging unit 201a is held near the user's right eye, and the first imaging unit 201b is held near the user's left eye. Based on such a configuration, the first imaging units 201a and 201b capture images of a subject (in other words, a real object located in real space) located in front of the input / output device 20 from different positions. This allows the input / output device 20 to obtain an image of the subject located in front of the user, and to calculate the distance from the input / output device 20 to the subject based on the parallax between the images captured by the first imaging units 201a and 201b, respectively.

[0019] Further, the second imaging units 203a and 203b are each held by the holding unit 291 so that the eyeballs of the user are located within the respective imaging ranges when the input / output device 20 is worn on the user's head. As a specific example, the second imaging unit 203a is held so that the right eye of the user is located within the imaging range. Based on such a configuration, it is possible to recognize the direction in which the line of sight of the right eye is directed based on the image of the eyeball of the right eye captured by the second imaging unit 203a and the positional relationship between the second imaging unit 203a and the right eye. Similarly, the second imaging unit 203b is held so that the left eye of the user is located within the imaging range. That is, it is possible to recognize the direction in which the line of sight of the left eye is directed based on the image of the eyeball of the left eye captured by the second imaging unit 203b and the positional relationship between the second imaging unit 203b and the left eye. In the following description, the direction in which the line of sight is directed is also referred to as the "line of sight direction". In addition, in the example shown in FIG. 1, the input / output device 20 includes both the second imaging units 203a and 203b, but only one of the second imaging units 203a and 203b may be provided.

[0020] The operation unit 207 is a component for receiving operations from a user to the input / output device 20. The operation unit 207 may be configured with an input device such as a touch panel or a button. The operation unit 207 is held at a predetermined position of the input / output device 20 by a holding unit 291. For example, in the example shown in FIG. 1, the operation unit 207 is held at a position corresponding to the temples of glasses.

[0021] Furthermore, the input / output device 20 according to the present embodiment may be provided with, for example, an acceleration sensor or an angular velocity sensor (gyro sensor) and configured to be capable of detecting the movement of the head of a user wearing the input / output device 20 (in other words, the movement of the input / output device 20 itself). As a specific example, the input / output device 20 may recognize a change in at least one of the position and posture of the user's head by detecting components in the yaw direction, pitch direction, and roll direction as the movement of the user's head.

[0022] Based on the above configuration, the input / output device 20 according to this embodiment can recognize changes in its own position and posture in real space according to the movement of the user's head. In addition, at this time, the input / output device 20 can also present virtual content (i.e., a virtual object) on the output unit 211 so that the content is superimposed on a real object located in the real space based on so-called AR technology. Note that an example of a method for the input / output device 20 to estimate its own position and posture in real space (i.e., self-position estimation) will be described in detail separately below.

[0023] Examples of a head-mounted display device (HMD) that can be used as the input / output device 20 include a see-through HMD, a video see-through HMD, and a retinal projection HMD.

[0024] The see-through HMD holds a virtual image optical system consisting of a transparent light guide unit, etc., in front of the user's eyes using, for example, a half mirror or a transparent light guide plate, and displays an image inside the virtual image optical system. Therefore, a user wearing the see-through HMD can view the outside scenery even while viewing an image displayed inside the virtual image optical system. With this configuration, the see-through HMD can also superimpose an image of a virtual object on an optical image of a real object located in a real space according to at least one of the recognition results of the position and the orientation of the see-through HMD based on, for example, AR technology. Note that a specific example of the see-through HMD is a so-called glasses-type wearable device in which a part equivalent to the lens of glasses is configured as a virtual image optical system. For example, the input / output device 20 shown in FIG. 1 corresponds to an example of a see-through HMD.

[0025] When the video see-through HMD is worn on the user's head or face, it is worn so as to cover the user's eyes, and a display unit such as a display is held in front of the user's eyes. The video see-through HMD also has an imaging unit for capturing images of the surrounding scenery, and displays an image of the scenery in front of the user captured by the imaging unit on the display unit. With this configuration, although it is difficult for a user wearing the video see-through HMD to directly view the outside scenery, the user can check the outside scenery from the image displayed on the display unit. In addition, the video see-through HMD may superimpose a virtual object on the image of the outside scenery according to at least one of the recognition results of the position and the orientation of the video see-through HMD based on AR technology, for example.

[0026] In the retinal projection HMD, a projection unit is held in front of the user's eyes, and an image is projected from the projection unit toward the user's eyes so that the image is superimposed on the external scenery. More specifically, in the retinal projection HMD, an image is directly projected from the projection unit onto the retina of the user's eye, and the image is formed on the retina. With this configuration, even in the case of a user with myopia or hyperopia, it becomes possible to view a clearer image. In addition, a user wearing a retinal projection HMD can view the external scenery while viewing an image projected from the projection unit. With this configuration, the retinal projection HMD can also superimpose an image of a virtual object on an optical image of a real object located in a real space according to at least one of the recognition results of the position and / or the posture of the retinal projection HMD based on AR technology, for example.

[0027] For reference, in addition to the examples described above, there is also an HMD called an immersive HMD. Like a video see-through HMD, an immersive HMD is worn over the user's eyes, and a display unit such as a display is held in front of the user's eyes. For this reason, it is difficult for a user wearing an immersive HMD to directly view the outside scenery (i.e., the scenery of the real world), and only the image displayed on the display unit comes into view. With this configuration, As a result, the immersive HMD can give a sense of immersion to the user viewing the image.

[0028] An example of a schematic configuration of an input / output device according to an embodiment of the present disclosure has been described above with reference to FIG.

[0029] <1.2. Principles of self-location estimation> Next, an example of the principle of a method for the input / output device 20 to estimate its own position and orientation in real space (that is, self-position estimation) when superimposing a virtual object on a real object will be described.

[0030] As a specific example of self-position estimation, the input / output device 20 captures an image of a marker or the like with a known size that is presented on a real object in the real space using an imaging unit such as a camera provided in the input / output device 20. The input / output device 20 then analyzes the captured image to estimate at least one of its own position and orientation relative to the marker (and thus the real object on which the marker is presented). Note that the following description focuses on a case where the input / output device 20 estimates its own position and orientation, but the input / output device 20 may estimate only one of its own position and orientation.

[0031] Specifically, it is possible to estimate the relative direction of the imaging unit (and thus the input / output device 20 including the imaging unit) with respect to the marker, depending on the orientation of the marker captured in the image (for example, the orientation of the pattern of the marker). In addition, when the size of the marker is known, it is possible to estimate the distance between the marker and the imaging unit (i.e., the input / output device 20 including the imaging unit) depending on the size of the marker in the image. More specifically, if the marker is captured from a greater distance, the marker will be captured smaller. In addition, the range of the real space captured in the image at this time can be estimated based on the angle of view of the imaging unit. By utilizing the above characteristics, it is possible to back-calculate the distance between the marker and the imaging unit depending on the size of the marker captured in the image (in other words, the proportion of the marker in the angle of view). With the above configuration, the input / output device 20 is able to estimate its own relative position and orientation with respect to the marker.

[0032] Also, a technology called SLAM (simultaneous localization and mapping) may be used for self-location estimation of the input / output device 20. SLAM is a technology that performs self-location estimation and the creation of an environmental map in parallel by using an imaging unit such as a camera, various sensors, an encoder, and the like. As a more specific example, in SLAM (particularly, Visual SLAM), a three-dimensional shape of an imaged scene (or a subject) is sequentially restored based on a moving image captured by the imaging unit. Then, the restoration result of the imaged scene is associated with the detection result of the position and orientation of the imaging unit, thereby creating a map of the surrounding environment and estimating the position and orientation of the imaging unit (and thus the input / output device 20) in the environment. Note that, for example, by providing various sensors such as an acceleration sensor and an angular velocity sensor in the input / output device 20, the position and orientation of the imaging unit can be estimated as information indicating a relative change based on the detection result of the sensor. Of course, as long as the position and orientation of the imaging unit can be estimated, the method is not necessarily limited to a method based on the detection result of various sensors such as an acceleration sensor and an angular velocity sensor.

[0033] With the above-mentioned configuration, for example, the estimation result of the relative position and orientation of the input / output device 20 with respect to a known marker based on the imaging result of the imaging unit may be used for the initialization process and position correction in the above-mentioned SLAM. With such a configuration, even in a situation where the marker is not included in the angle of view of the imaging unit, the input / output device 20 can estimate the position of the marker (or a marker) by self-position estimation based on SLAM that has received the results of the initialization and position correction previously performed. In this case, it becomes possible to estimate the position and orientation of the marker relative to the real object on which it is presented.

[0034] An example of the principle of a method (i.e., self-location estimation) for estimating the position and orientation of the input / output device 20 in real space when the input / output device 20 superimposes a virtual object on a real object has been described above. Note that, hereinafter, for example, a description will be given assuming that the position and orientation of the input / output device 20 relative to an object (real object) in real space can be estimated based on the above-mentioned principle.

[0035] <<2. First embodiment>> Next, an example of an information processing system according to a first embodiment of the present disclosure will be described. The information processing system according to this embodiment applies the above-mentioned AR and self-location estimation (SLAM, etc.) technologies, making it possible to measure distances between multiple positions in real space based on operations via the input / output device 20. Therefore, hereinafter, the information processing system according to this embodiment will be described in terms of a distance measurement method using the system, the configuration of the system, and the processing of the system.

[0036] <2.1.Measurement method> First, an example of a method for measuring distance in real space using the information processing system according to this embodiment will be described with reference to Figures 2 and 3. For example, Figure 2 is an explanatory diagram for explaining an overview of the information processing system according to this embodiment, and shows an example of a method for measuring distance in real space. Note that Figure 2 presents both real objects and virtual objects in order to make the features of the information processing system according to this embodiment easier to understand.

[0037] In the information processing system according to this embodiment, the user wears the input / output device 20 and then specifies the start and end points of distance measurement in real space by the line of sight. At this time, the system estimates the three-dimensional positions of the start and end points specified on the real object 90 based on the measurement result of the distance between the input / output device 20 and the real object 90 and the recognition result of the user's line of sight. More specifically, the system recognizes the intersection point of a mesh surface based on the measurement result of the distance (depth) between the input / output device 20 and the real object 90 and a vector indicating the line of sight direction (hereinafter also referred to as a "line of sight vector") as the point on the real object 90 specified by the user (i.e., the start or end point).

[0038] As a specific procedure, first, the user wears the input / output device 20, and performs a predetermined operation (e.g., an operation on the operation unit 207) for registering the start point while gazing at a position on the real object 90 that is to be the start point of distance measurement. In response to this operation, the system estimates the three-dimensional position in real space where the user is gazing, and registers the estimated position as the start point. In addition, at this time, the system may present a virtual object V11 indicating the start point to the user via the input / output device 20 (e.g., the output unit 211) so that the virtual object V11 is superimposed on the position in real space registered as the start point.

[0039] After the registration of the start point is completed, the user performs a predetermined operation for registering the end point while gazing at the position on the real object 90 that is to be the end point of the distance measurement. In response to this operation, the system registers the end point in the same manner as in the case of the start point. At this time, the system may present the virtual object V13 indicating the end point to the user via the input / output device 20 so that the virtual object V13 is superimposed on the position in real space registered as the end point.

[0040] The system then estimates the three-dimensional positions of the registered start and end points. Based on this, the distance between the start point and the end point is measured (calculated), and display information V17 indicating the measurement result is presented to the user via the input / output device 20 (e.g., the display unit 211). For example, FIG. 3 is a diagram showing an example of display information presented according to the distance measurement result. With this configuration, the user can measure the distance in real space by a simpler operation using the input / output device 20, without actually using a tool such as a ruler or tape measure to measure the distance.

[0041] In addition, at this time, the system may present display information indicating a scale (dimension) in real space. For example, in the example shown in FIG. 2 and FIG. 3, after the start point is registered, the system presents tape-measure-shaped virtual objects V15a and V15b between the start point and the position where the user's gaze is directed in response to the movement of the user's gaze. Specifically, the system presents a virtual object V15a corresponding to a tape measure case to indicate the position in real space where the user's gaze is directed. In addition, the system presents a band-shaped virtual object V15b with a scale according to the dimension in real space between the registered start point and the virtual object V15a. With this control, the user can measure the distance in real space by operating the input / output device 20 with the same feeling as when actually measuring the distance using a tool such as a ruler or a tape measure. In the following description, the position where the user's gaze is directed is referred to as a "gazing point," and the position that is the base point of the gaze (for example, a position corresponding to the eye, in other words, the position of the input / output device 20) may be referred to as a "viewpoint."

[0042] An example of a method for measuring a distance in real space using the information processing system according to this embodiment has been described above with reference to FIGS.

[0043] <2.2. Functional configuration> Next, an example of the functional configuration of the information processing system according to the present embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the functional configuration of the information processing system according to the present embodiment.

[0044] 4, the information processing system 1 according to the present embodiment includes, for example, an input / output device 20 and an information processing device 10. The input / output device 20 corresponds to the input / output device 20 described with reference to FIG.

[0045] First, the configuration of the input / output device 20 will be described. As shown in FIG. 4, the input / output device 20 includes a first imaging unit 201, a second imaging unit 203, an input unit 205, and an output unit 211. The input unit 205 includes an operation unit 207. The input unit 205 may include a sound collection unit 209. The first imaging unit 201 corresponds to the first imaging units 201a and 201b shown in FIG. 1. The second imaging unit 203 corresponds to the second imaging units 203a and 203b shown in FIG. 1. The operation unit 207 and the output unit 211 correspond to the operation unit 207 and the output unit 211 shown in FIG. 1, respectively.

[0046] The first imaging unit 201 is configured as a so-called stereo camera, and captures images of a subject from a plurality of different positions, i.e., an image of the subject from a viewpoint corresponding to the right eye side and an image of the subject from a viewpoint corresponding to the left eye side. In the following description, an image captured from a viewpoint corresponding to the right eye side is also referred to as a "right eye image", and an image captured from a viewpoint corresponding to the left eye side is also referred to as a "left eye image". The first imaging unit 201 then outputs the captured right eye image and left eye image to the information processing device 10.

[0047] The second imaging unit 203 captures an image of the eyeball of the user wearing the input / output device 20, and outputs the captured image to the information processing device 10. Note that, at this time, the second imaging unit 203 captures an image of each of the right and left eyes of the user, and outputs each of the images to the information processing device 10. You may do so.

[0048] When the operation unit 207 receives an operation from the user, it outputs control information indicating the content of the operation to the information processing device 10. In addition, the sound collection unit 209 collects sound such as the user's voice, and outputs an audio signal based on the sound collection result to the information processing device 10.

[0049] The output unit 211 is configured by a so-called display or the like, and displays display information such as an image based on the control from the information processing device 10. For example, as described above with reference to FIG. 1, the output unit 211 may be configured as a so-called transmissive display.

[0050] Next, the description will be focused on the configuration of the information processing device 10. As shown in Fig. 4, the information processing device 10 includes a depth detection unit 101, a position and orientation estimation unit 103, a gaze detection unit 105, a gaze point detection unit 107, a trigger detection unit 109, a reference point estimation unit 111, a distance calculation unit 113, and an output control unit 115.

[0051] The depth detection unit 101 acquires the right eye image and the left eye image captured by the first imaging unit 201 from the first imaging unit 201. The depth detection unit 101 calculates the distance (i.e., depth) between the input / output device 20 and the subject captured in the right eye image and the left eye image based on the parallax between the right eye image and the left eye image. Then, the depth detection unit 101 outputs the acquired image of the subject (e.g., the right eye image and the left eye image) and depth information indicating the calculated depth (e.g., information indicating a mesh surface based on the depth measurement result) to the position and orientation estimation unit 103. The depth detection unit 101 also outputs the depth information to the gaze point detection unit 107.

[0052] The position and orientation estimation unit 103 is a component for executing processing relating to estimation of the position and orientation of the input / output device 20 in the real space (processing relating to so-called self-position estimation).

[0053] Specifically, the position and orientation estimation unit 103 acquires an image of a subject and depth information from the depth detection unit 101. The position and orientation estimation unit 103 performs an analysis process on each acquired image to recognize an object (real object) captured as a subject in the image. At this time, the position and orientation estimation unit 103 may recognize the object captured in the image by calculating a feature amount based on the features (e.g., features such as shape and pattern) of the object captured in the image and comparing the feature amount with information indicating the feature amount of a known object.

[0054] Next, the position and orientation estimation unit 103 estimates the relative position and orientation of the input / output device 20 with respect to the real object based on the recognition result of the real object captured in the acquired image and the acquired depth information (i.e., information indicating the distance between the input / output device 20 and the real object). At this time, the position and orientation estimation unit 103 may estimate the relative position and orientation of the input / output device 20 with respect to the recognized real object based on SLAM. In this case, the position and orientation estimation unit 103 may acquire information indicating changes in the position and orientation of the input / output device 20 from a predetermined detection unit (e.g., an acceleration sensor or an angular velocity sensor provided in the input / output device 20) not shown, and use the information for self-position estimation based on SLAM (i.e., estimation of the position and orientation of the input / output device 20 with respect to the real object).

[0055] Then, the position and orientation estimation unit 103 outputs information indicating the estimation result of the position and orientation of the input / output device 20 in the real space to the reference point estimation unit 111. In addition, the position and orientation estimation unit 103 outputs the recognition result of the real object and information indicating the estimation result of the position and orientation of the input / output device 20 in the real space to the output control unit 115.

[0056] The gaze detection unit 105 acquires an image of the user's eyeball captured by the second imaging unit 203 from the second imaging unit 203. The gaze detection unit 105 performs an analysis process on the acquired image. By carrying out this process, the direction in which the eyeball captured in the image is facing is recognized. The gaze detection unit 105 detects the direction in which the eyeball is facing in real space, that is, the gaze direction, based on the recognition result of the direction in which the eyeball captured in the image is facing and the positional relationship between the eyeball and the second imaging unit 203. It goes without saying that the positional relationship between the eyeball and the second imaging unit 203 can be recognized or estimated in advance based on the assumed wearing state of the input / output device 20. The gaze direction detected at this time corresponds to a relative direction based on the position and posture of the input / output device 20. Then, the gaze detection unit 105 outputs gaze information indicating the detection result of the gaze direction to the gaze point detection unit 107.

[0057] The gaze point detection unit 107 acquires depth information indicating a result of depth detection from the depth detection unit 101. The gaze point detection unit 107 also acquires gaze information indicating a result of gaze detection from the gaze detection unit 105. Then, based on the acquired depth information and gaze information, the gaze point detection unit 107 detects a position in real space to which the user's gaze is directed (i.e., the position of the gaze point in real space) as a relative position based on the position and attitude of the input / output device 20.

[0058] Specifically, the gaze point detection unit 107 calculates a gaze vector based on the gaze information, and detects the intersection of the gaze vector with a mesh surface based on the depth information (i.e., three-dimensional position information of the surface of a real object) as the position of the gaze point in real space. Note that the position detected at this time is a relative position based on the position and attitude of the input / output device 20, as described above. Then, the gaze point detection unit 107 outputs the detected position, i.e., information indicating the position of the gaze point in real space (hereinafter also referred to as "position information of the gaze point"), to the reference point estimation unit 111.

[0059] The trigger detection unit 109 acquires information indicating a user input via the input unit 205 from the input unit 205. Then, when the acquired information indicating the user input indicates a predetermined operation content, the trigger detection unit 109 uses the user input as a trigger to give an instruction associated with the operation content to the reference point estimation unit 111. As a specific example, when the acquired information indicating the user input indicates registration of a position in real space that serves as a reference for distance measurement, such as the above-mentioned start point or end point (hereinafter also referred to as a "reference point"), the trigger detection unit 109 uses the user input as a trigger to instruct the reference point estimation unit 111 to register the reference point.

[0060] As a specific example, the trigger detection unit 109 acquires control information indicating the operation content from the operation unit 207, and when the control information indicates an operation for registering a reference point, the trigger detection unit 109 uses the operation as a trigger to instruct the reference point estimation unit 111 to register a reference point.

[0061] As another example, the trigger detection unit 109 may acquire an acoustic signal based on the sound collection result from the sound collection unit 209. In this case, the trigger detection unit 109 recognizes the contents of the user's utterance by performing various analysis processes based on so-called voice recognition processing and natural etymology processing on the acquired acoustic signal. Then, when the contents of the user's utterance indicate the registration of a reference point, the trigger detection unit 109 may instruct the reference point estimation unit 111 to register a reference point using the recognition result of the utterance contents as a trigger.

[0062] The reference point estimation unit 111 acquires information indicating an estimation result of the position and orientation of the input / output device 20 in real space from the position / orientation estimation unit 103. The reference point estimation unit 111 also acquires position information of the gaze point from the gaze point detection unit 107. Then, when the reference point estimation unit 111 receives an instruction for registering a reference point from the trigger detection unit 109, the reference point estimation unit 111 estimates the position of a reference point, such as a start point or an end point, in real space based on the estimation result of the position and orientation of the input / output device 20 and the position information of the gaze point.

[0063] Specifically, the reference point estimation unit 111 uses the position information of the gaze point acquired from the gaze point detection unit 107 as Based on the information acquired from the position and orientation estimation unit 103, the reference point estimation unit 111 recognizes the three-dimensional position of the gaze point in real space as a relative position based on the position and orientation of the input / output device 20. Furthermore, the reference point estimation unit 111 recognizes the position and orientation of the input / output device 20 in real space based on the information acquired from the position and orientation estimation unit 103. This enables the reference point estimation unit 111 to estimate the three-dimensional position of the gaze point in real space as, for example, an absolute position.

[0064] Then, the reference point estimation unit 111 registers a reference point such as a start point or an end point based on an instruction from the trigger detection unit 109 to register the reference point. For example, when the reference point estimation unit 111 receives an instruction to register the reference point, the reference point estimation unit 111 may register the start point or the end point depending on the registration status of the reference point. More specifically, when the reference point estimation unit 111 receives the instruction, if the start point has not been registered, the reference point estimation unit 111 registers the reference point estimated at that time (i.e., the position of the gaze point in the real space) as the start point. Also, when the reference point estimation unit 111 receives the instruction, if the start point has already been registered, the reference point estimation unit 111 may register the reference point estimated at that time as the end point.

[0065] As another example, the reference point estimation unit 111 may receive instructions for registration of the start point and the end point separately from the trigger detection unit 109. For example, when the reference point estimation unit 111 recognizes that the user has instructed to register the start point based on an instruction from the trigger detection unit 109, the reference point estimation unit 111 registers the reference point estimated at that time as the start point. Similarly, when the reference point estimation unit 111 recognizes that the user has instructed to register the end point based on an instruction from the trigger detection unit 109, the reference point estimation unit 111 may register the reference point estimated at that time as the end point.

[0066] In this manner, the reference point estimation unit 111 registers reference points such as a start point and an end point based on an instruction from the trigger detection unit 109, and outputs position information indicating the position in real space of each of the registered reference points to the distance calculation unit 113. This enables the distance calculation unit 113 to recognize the position in real space of each registered reference point.

[0067] Furthermore, the reference point estimation unit 111 may sequentially output information indicating the position of the gaze point in real space to the distance calculation unit 113. This enables the distance calculation unit 113 to recognize the position in real space to which the user's gaze is directed (i.e., the position of the gaze point in real space) in real time.

[0068] The distance calculation unit 113 acquires position information indicating the position of each registered reference point (e.g., a start point and an end point) in real space from the reference point estimation unit 111. The distance calculation unit 113 calculates the distance between a plurality of reference points based on the position information of each registered reference point, and outputs information indicating the calculation result of the distance to the output control unit 115. In addition, at this time, the distance calculation unit 113 may output information indicating the position information of each registered reference point to the output control unit 115.

[0069] Furthermore, the distance calculation unit 113 may sequentially acquire information indicating the position of the gaze point in real space from the reference point estimation unit 111. In this case, the distance calculation unit 113 may, for example, calculate the distance between a registered reference point (for example, a start point) and the gaze point, and output information indicating the calculation result of the distance to the output control unit 115. At this time, the distance calculation unit 113 may also output information indicating the position of the gaze point in real space to the output control unit 115.

[0070] The output control unit 115 presents the virtual object to the user via the output unit 211 based on AR technology so that the virtual object is superimposed on the real space.

[0071] Specifically, the output control unit 115 acquires, from the position and orientation estimation unit 103, information indicating the recognition result of the real object and the estimation result of the position and orientation of the input / output device 20 in the real space. This enables the output control unit 115 to estimate the positional relationship between the input / output device 20 and the recognized real object in the real space.

[0072] Then, the output control unit 115 causes the output unit 211 to display the virtual object so that the virtual object is superimposed at a desired position in the real space according to the positional relationship in the real space between the input / output device 20 and the recognized real object. At this time, the output control unit 115 may execute various processes related to the generation of display information, such as rendering, in order to present the virtual object as CG (Computer Graphics).

[0073] Furthermore, the output control unit 115 may acquire information indicating the calculation result of the distance from the distance calculation unit 113, and cause the output unit 211 to display display information corresponding to the calculation result of the distance based on the information. At this time, the output control unit 115 may acquire position information of each registered reference point from the distance calculation unit 113. In this case, the output control unit 115 may cause the output unit 211 to display the display information indicating each reference point based on the acquired position information so that the display information is superimposed on the position of the reference point in real space.

[0074] Furthermore, the output control unit 115 may acquire information indicating a calculation result of the distance between the registered reference point and the gaze point from the distance calculation unit 113, and cause the output unit 211 to display display information corresponding to the calculation result of the distance based on the acquired information. At this time, the output control unit 115 may acquire information indicating the position of the gaze point in real space from the distance calculation unit 113. In this case, the output control unit 115 may cause the output unit 211 to display, for example, display information (for example, virtual objects V15a and V15b as shown in FIG. 2 and FIG. 3) corresponding to the respective positional relationships between the registered reference point and the gaze point.

[0075] It should be noted that the configuration shown in Fig. 4 is merely an example, and the configuration of the information processing system 1 is not necessarily limited to the example shown in Fig. 4. As a specific example, the input / output device 20 and the information processing device 10 may be integrally configured. As another example, some of the configurations of the information processing device 10 may be provided in a device different from the information processing device 10 (for example, the input / output device 20 or an external server, etc.).

[0076] In the above description, the input / output device 20 is configured as a so-called see-through HMD as shown in FIG. 1, but as described above, the input / output device 20 may be configured as a video see-through HMD or a retinal projection HMD. In that case, it goes without saying that a part of the configuration and processing of the information processing device 10 may be replaced as necessary. As a specific example, when the input / output device 20 is configured as a video see-through HMD, the output control unit 115 may cause the output unit 211 to display an image in which a virtual object is superimposed on an image captured by the first imaging unit 201.

[0077] In the above-described example, an example has been described in which an image of an object (real object) in real space and depth information indicating the distance from the input / output device 20 to the object are acquired based on an image captured by a stereo camera. On the other hand, as long as it is possible to acquire an image of an object in real space and depth information indicating the distance from the input / output device 20 to the object, the configuration for acquiring the image and the depth information is not particularly limited. For example, a distance measuring unit for measuring the distance may be provided in addition to an imaging unit for acquiring an image of an object in real space. It goes without saying that the configuration of the distance measuring unit is not particularly limited. As a more specific example, the distance from the input / output device 20 to the object may be measured based on a method such as motion parallax, TOF (Time Of Flight), or Structured Light.

[0078] Here, TOF refers to projecting infrared light or other light onto a subject, measuring the time it takes for the light to reflect off the subject and return for each pixel, and then calculating the time it takes for the light to reflect off the subject and return. Structured Light is a method of obtaining an image (so-called distance image) including the distance (depth) to the subject based on the change in the pattern obtained from the imaging result by irradiating a pattern on the subject with light such as infrared light and imaging it, and obtaining a distance image including the distance (depth) to the subject based on the change in the pattern obtained from the imaging result. Moving parallax is a method of measuring the distance to the subject based on parallax even in a so-called monocular camera. Specifically, the camera is moved to image the subject from different viewpoints, and the distance to the subject is measured based on the parallax between the captured images. At this time, the distance and direction of movement of the camera are recognized by various sensors, making it possible to measure the distance to the subject with higher accuracy. The configuration of the imaging unit (for example, monocular camera, stereo camera, etc.) may be changed depending on the distance measurement method.

[0079] An example of the functional configuration of the information processing system according to the present embodiment has been described above with reference to FIG.

[0080] <2.3. Processing> Next, an example of a series of process flows of the information processing system according to the present embodiment will be described with reference to Fig. 5, focusing in particular on the process of the information processing device 10. Fig. 5 is a flowchart showing an example of a series of process flows of the information processing system according to the present embodiment.

[0081] First, the information processing device 10 (depth detection unit 101) calculates the distance (depth) between the input / output device 20 and the captured subject (i.e., a real object) based on the right-eye image and the left-eye image captured by the first imaging unit 201. As a result, an image of the real object and depth information indicating the distance between the input / output device 20 and the real object are acquired (S101).

[0082] Next, the information processing device 10 (position and orientation estimation unit 103) estimates the relative position and orientation of the input / output device 20 with respect to the real object captured in the image based on the acquired image and depth information. At this time, the position and orientation estimation unit 103 may also estimate the relative position and orientation of the input / output device 20 with respect to the recognized real object based on SLAM (S103). Note that the method of estimating the relative position and orientation of the input / output device 20 is as described above as the processing of the position and orientation estimation unit 103.

[0083] In addition, the information processing device 10 (gaze direction detection unit 105) detects the gaze direction based on the image of the user's eyeball captured by the second imaging unit 203 and the positional relationship between the eyeball and the second imaging unit 203 (S105).

[0084] Next, based on the detection result of the gaze direction and the acquired depth information, the information processing device 10 (gazing point detection unit 107) detects the position in real space to which the user's gaze is directed (i.e., the position of the gaze point in real space) as a relative position based on the position and orientation of the input / output device 20. Furthermore, based on the detection result of the position of the gaze point in real space and the estimation result of the position and orientation of the input / output device 20 in real space, the information processing device 10 (reference point estimation unit 111) estimates the three-dimensional position of the gaze point in real space as an absolute position (S107).

[0085] The information processing device 10 sequentially executes a series of processes indicated by reference signs S101 to S107 unless a predetermined trigger based on a user input via the input unit 205 is detected (S109, NO). Then, when a predetermined trigger based on a user input is detected (S109, YES), the information processing device 10 executes processes related to registration of the start point and the end point.

[0086] For example, when the start point is not registered (S111, NO), the information processing device 10 (reference point estimation unit 111) estimates the three-dimensional position of the gaze point in the real space when the trigger is detected. After the start point is registered, the information processing device 10 (distance calculation unit 113) calculates the distance from the registered start point to the current gaze point (S115).

[0087] Furthermore, when the start point has already been registered (S111, YES), the information processing device 10 (reference point estimation unit 111) registers the three-dimensional position in real space of the gaze point when the trigger is detected as the end point (S117). In this case, the information processing device 10 (distance calculation unit 113) calculates the distance from the registered start point to the end point. Then, the information processing device 10 (output control unit 115) causes the output unit 211 of the input / output device 20 to display information based on the calculation result of the distance.

[0088] The information processing device 10 executes the above-mentioned series of processes until the user instructs the end of the process (for example, the end of the application) (S121, NO). Then, when the user instructs the end of the process (S121, YES), the information processing device 10 ends the above-mentioned series of processes.

[0089] An example of the flow of a series of processes in the information processing system according to the present embodiment has been described above with reference to FIG.

[0090] <2.4. Evaluation> As described above, in the information processing system according to the present embodiment, the user wears the input / output device 20 and then specifies the start and end points of distance measurement in real space by the line of sight. At this time, the information processing device 10 estimates the positions of reference points such as the start and end points in real space based on each image captured by the input / output device 20 from a plurality of different viewpoints and the estimation results of the position and attitude of the input / output device 20 at each viewpoint. In addition, the information processing device 10 calculates (measures) the distance between the start and end points based on the estimation results of the positions of the start and end points in real space. Then, the information processing device 10 presents display information based on the calculation results of the distance between the start and end points to the user via the input / output device 20. With this configuration, the user can measure the distance in real space by a simpler operation using the input / output device 20 without actually using a tool such as a ruler or tape measure to measure the distance.

[0091] In addition, in the information processing system according to the present embodiment, the position and orientation of the input / output device 20 are sequentially estimated based on a self-position estimation technique such as SLAM during a series of operations related to measuring distances in real space. With this configuration, when each reference point such as a start point or an end point is registered, for example, it is possible to estimate the position of the reference point in real space based on the position and orientation of the input / output device 20 at the time of registration (in other words, the viewpoint at the time of registration). Therefore, according to the system according to the present embodiment, even if it is difficult to simultaneously capture each position in real space registered as a start point and an end point in an image captured by the input / output device 20, it is possible to measure the distance between the start point and the end point. Note that, examples of situations in which it is difficult to simultaneously capture each position in real space registered as a start point and an end point in an image include a case where the start point and the end point are relatively far apart, a case where an obstruction exists between the start point and the end point, and the like.

[0092] In the above example, an example of the case where the distance between the start point and the end point is measured by registering the start point and the end point as reference points has been described, but the present invention is not necessarily limited to this embodiment. As a specific example, the information processing system according to this embodiment may be configured to be able to register three or more reference points and to be able to measure the distance between each of the three or more registered reference points. As a more specific example, when a new end point is registered after an end point is registered, the information processing device 10 may set the previously registered end point as a via point. Then, the distance between the start point and the waypoint and the distance between the waypoint and the newly registered end point may be calculated. The number of waypoints is not limited to one, and multiple waypoints may be registered. This also applies to information processing systems according to other embodiments and modified examples described later.

[0093] <<3. Second embodiment>> Next, an example of an information processing system according to a second embodiment of the present disclosure will be described.

[0094] <3.1.Measurement method> First, an example of a method for measuring distance in real space using the information processing system according to this embodiment will be described with reference to Fig. 6. For example, Fig. 6 is an explanatory diagram for explaining an overview of the information processing system according to this embodiment, and shows an example of a method for measuring distance in real space. Note that Fig. 6 presents both real objects and virtual objects in order to make the features of the information processing system according to this embodiment easier to understand.

[0095] The information processing system according to this embodiment detects the line of sight of a user wearing an input / output device 20, and presents a virtual object V19 for specifying a reference point such as a start point or an end point to the user via the input / output device 20 based on the detection result of the line of sight. For example, in the example shown in FIG. 6, arrow-shaped virtual objects V19a and V19b indicating the line of sight of the user are presented. The user specifies the start point and end point of distance measurement in real space by adjusting the direction and length of the virtual object V19 presented according to the detection result of the user's line of sight based on an operation via the input / output device 20.

[0096] Specifically, as indicated by reference symbol P11, the user first wears the input / output device 20, gazes at a position in real space to be registered as a starting point, and performs a predetermined operation related to the placement of a virtual object V19 on the input / output device 20. In response to the operation, the system detects the user's line of sight, and presents a virtual object V19a to the user via the input / output device 20 so that the virtual object V19a according to the detected line of sight direction (i.e., line of sight vector) is superimposed on the real space.

[0097] Next, as shown by reference symbol P12, the user refers to the virtual object V19a presented so as to be superimposed on the real space from another viewpoint (e.g., another angle), and adjusts the direction and length of the virtual object V19a by operating via the input / output device 20. By such an operation, the user specifies a position in the real space to be registered as a starting point by the virtual object V19a. In response to this operation, the system registers the position in the real space specified by the virtual object V19a as the starting point. For example, in the example shown in FIG. 6, the system registers the position in the real space pointed to by the arrow-shaped virtual object V19a as the starting point.

[0098] The user also registers the end point in the same manner as the start point. For example, as shown by reference symbol P13, the user gazes at the position in real space to be registered as the end point and performs a predetermined operation. In response to the operation, the system detects the user's line of sight and presents a virtual object V19b to the user according to the detection result. The user also refers to the presented virtual object V19b from another viewpoint and adjusts the direction and length of the virtual object V19b by operating via the input / output device 20, thereby specifying the position in real space to be registered as the end point. Then, the system registers the position in real space indicated by the arrow-shaped virtual object V19b as the end point.

[0099] The operation after the start point and the end point are registered is the same as that of the first embodiment. That is, the system performs the following operations based on the three-dimensional positions of the registered start point and the end point: The distance between the start point and the end point is measured (calculated), and various display information according to the measurement result is presented to the user via the input / output device 20. With this configuration, the user can measure the distance in real space by a simpler operation using the input / output device 20, without actually using a tool such as a ruler or tape measure to measure the distance.

[0100] Furthermore, in the information processing system according to this embodiment, positions in real space to be registered as reference points such as a start point and an end point are specified by adjusting the orientation and length of the virtual object V19. Therefore, in the information processing system according to this embodiment, unlike the information processing system according to the first embodiment, it is also possible to register a position in real space where no real object exists as a reference point.

[0101] An example of a method for measuring a distance in real space using the information processing system according to this embodiment has been described above with reference to FIG.

[0102] <3.2. Processing> Next, with reference to Figs. 7 to 9, an example of a series of process flows of the information processing system according to this embodiment will be described, focusing in particular on the process of the information processing device 10. Figs. 7 to 9 are flowcharts showing an example of a series of process flows of the information processing system according to this embodiment. Note that the basic functional configuration of the information processing system according to this embodiment is similar to that of the system according to the first embodiment described above (see Fig. 4), but the process in some components is different. Therefore, in this description, the features of the information processing system according to this embodiment will be described in more detail, focusing in particular on the process that is different from that of the information processing system according to the first embodiment described above.

[0103] The processes indicated by reference signs S101 to S107 are the same as those in the information processing system according to the first embodiment described above (see FIG. 5). That is, the information processing device 10 (depth detection unit 101) acquires an image of a real object and depth information indicating a distance between the input / output device 20 and the real object based on the right eye image and the left eye image captured by the first imaging unit 201 (S101). Next, the information processing device 10 (position and orientation estimation unit 103) estimates the relative position and orientation of the input / output device 20 with respect to the real object captured in the image based on the acquired image and depth information (S103). In addition, the information processing device 10 (gaze detection unit 105) detects the gaze direction based on the image of the user's eyeball captured by the second imaging unit 203 and the positional relationship between the eyeball and the second imaging unit 203 (S105). Furthermore, the information processing device 10 (gazing point detection unit 107) detects the position of the gaze point in real space as a relative position based on the position and orientation of the input / output device 20 based on the detection result of the gaze direction and the acquired depth information. Then, the information processing device 10 (reference point estimation unit 111) estimates the three-dimensional position of the gaze point in real space as an absolute position based on the detection result of the position of the gaze point in real space and the estimation result of the position and orientation of the input / output device 20 in real space (S107). At this time, the information processing device 10 (reference point estimation unit 111) may also estimate the three-dimensional position and orientation in real space of a vector (in other words, a gaze vector) connecting the input / output device 20 and the gaze point as an absolute position.

[0104] The information processing device 10 sequentially executes a series of processes indicated by reference signs S101 to S107 unless a predetermined trigger based on a user input via the input unit 205 is detected (S123, NO and S125, NO).

[0105] Furthermore, when the information processing device 10 detects a trigger for adjusting the reference point (S123, YES), it executes each process related to the adjustment of the reference point such as the start point and the end point (S150). For example, Fig. 8 is a flowchart showing an example of the flow of a series of processes related to the adjustment of the reference point.

[0106] Specifically, the information processing device 10 (display control unit 115) presents a virtual object V19 for specifying a reference point to the user via the output unit 211 of the input / output device 20, depending on the three-dimensional position and orientation in real space of a vector connecting the input / output device 20 and the gaze point (S151).

[0107] Next, when the start point is not registered (S153, NO), the information processing device 10 (reference point estimation unit 111) adjusts the three-dimensional position of the reference point registered as the start point according to the adjustment result of the orientation and length of the virtual object V19 based on the user input. That is, the information processing device 10 recognizes the three-dimensional position indicated by the virtual object V19 whose orientation and length have been adjusted as the position of the reference point after adjustment (S155). After adjusting the position of the reference point registered as the start point, the information processing device 10 (distance calculation unit 113) calculates the distance from the reference point whose position has been adjusted to the current gaze point (S157).

[0108] Furthermore, when the start point has already been registered (S153, YES), the information processing device 10 (reference point estimation unit 111) adjusts the three-dimensional position of the reference point registered as the end point according to the adjustment result of the orientation and length of the virtual object V19 based on the user input. That is, the information processing device 10 recognizes the three-dimensional position indicated by the virtual object V19 whose orientation and length have been adjusted as the position of the reference point after adjustment (S159). In this case, the information processing device 10 (distance calculation unit 113) calculates the distance from the start point to the reference point whose position has been adjusted (i.e., the reference point registered as the end point), and causes the output unit 211 of the input / output device 20 to display information based on the calculation result of the distance.

[0109] 7, when the information processing device 10 detects a trigger for registering a reference point (S123, NO and S125, YES), it executes processing related to registering reference points such as a start point and an end point and measuring a distance (S170). For example, FIG. 9 is a flowchart showing an example of a series of processing flow related to registering a reference point and measuring a distance.

[0110] Specifically, when the start point is not registered (S171, NO), the information processing device 10 (reference point estimation unit 111) registers the three-dimensional position in real space of the gaze point when the trigger is detected (i.e., the position pointed to by the virtual object V19) as the start point (S173). After registering the start point, the information processing device 10 (distance calculation unit 113) calculates the distance from the registered start point to the current gaze point (S175).

[0111] Furthermore, when the start point has already been registered (S171, YES), the information processing device 10 (reference point estimation unit 111) registers the three-dimensional position of the gaze point in real space when the trigger is detected (i.e., the position pointed to by the virtual object V19) as the end point (S177). In this case, the information processing device 10 (distance calculation unit 113) calculates the distance from the registered start point to the end point, and causes the output unit 211 of the input / output device 20 to display information based on the calculation result of the distance.

[0112] 7, the information processing device 10 executes the above-mentioned series of processes until the user instructs the end of the process (for example, the end of the application) (S127, NO). When the user instructs the end of the process (S127, YES), the information processing device 10 ends the above-mentioned series of processes.

[0113] An example of a series of processing flows in the information processing system according to the present embodiment has been described above with reference to FIGS.

[0114] <3.3. Evaluation> As described above, in the information processing system according to the present embodiment, the information processing device 10 detects the line of sight of the user wearing the input / output device 20, and presents the virtual object V19 for specifying reference points such as a start point and an end point to the user via the input / output device 20 based on the detection result of the line of sight. The user adjusts the direction and length of the virtual object V19 presented according to the detection result of the user's line of sight based on the operation via the input / output device 20 to specify the start point and the end point of the distance measurement in the real space. The information processing device 10 recognizes the three-dimensional positions of the start point and the end point according to the position in the real space indicated by the virtual object V19 whose direction and length have been adjusted, and calculates (measures) the distance between the start point and the end point. Then, the information processing device 10 presents the display information based on the calculation result of the distance between the start point and the end point to the user via the input / output device 20. With such a configuration, the user can measure the distance in the real space by a simpler operation using the input / output device 20 without actually using a tool such as a ruler or a tape measure to measure the distance.

[0115] Furthermore, in the information processing system according to this embodiment, positions in real space to be registered as reference points such as a start point and an end point are specified by adjusting the orientation and length of the virtual object V19. Therefore, in the information processing system according to this embodiment, unlike the information processing system according to the first embodiment, it is also possible to register a position in real space where no real object exists as a reference point.

[0116] <<4. Third embodiment>> Next, an example of an information processing system according to a third embodiment of the present disclosure will be described.

[0117] <4.1.Measurement method> First, an example of a method for measuring distance in real space using the information processing system according to this embodiment will be described with reference to Fig. 10 and Fig. 11. For example, Fig. 10 and Fig. 11 are explanatory diagrams for explaining an overview of the information processing system according to this embodiment, and show an example of a method for measuring distance in real space. Note that Fig. 10 and Fig. 11 present both real objects and virtual objects together in order to make the features of the information processing system according to this embodiment easier to understand.

[0118] For example, as shown in FIG. 10, in the information processing system according to this embodiment, a user specifies the start and end points of distance measurement in real space by touching or bringing the input / output device 20 into close proximity with a desired position on a real object 90.

[0119] Specifically, as shown by reference symbol P21, the user first brings the input / output device 20 into contact with or into proximity with a position in real space to be registered as a start point, and performs a predetermined operation related to registering the start point on the input / output device 20. In response to the operation, the system estimates the three-dimensional position in real space where the input / output device 20 has come into contact with or been in proximity with, based on the position and orientation of the input / output device 20 and the distance (depth) between the input / output device 20 and the real object 90, and registers the estimated position as the start point.

[0120] The registration of an end point is similar to that of a start point. For example, as shown by reference symbol P22, the user brings the input / output device 20 into contact with or into close proximity to a position in real space to be registered as an end point, and performs a predetermined operation for registering the end point on the input / output device 20. In response to the operation, the system estimates the three-dimensional position in real space where the input / output device 20 has come into contact with or been close to, based on the position and orientation of the input / output device 20 and the distance (depth) between the input / output device 20 and the real object 90, and registers the estimated position as the end point.

[0121] The operations after the start point and end point are registered are the same as those in the above-described embodiments. That is, the system measures (calculates) the distance between the registered start point and end point based on the three-dimensional positions of the start point and end point, and presents various display information according to the measurement results to the user via the input / output device 20. With this configuration, the user can measure the distance in real space by simpler operations using the input / output device 20, without actually using tools such as a ruler or tape measure to measure the distance.

[0122] In the system according to the present embodiment, a terminal device 30 such as a smartphone or tablet terminal may be used instead of the input / output device 20 as a device for measuring a distance in real space. For example, FIG. 11 shows an overview of a procedure for measuring a distance in real space using the terminal device 30 in the information processing system according to the present embodiment. Note that in the information processing system according to the present embodiment, the terminal device 30 is configured to be capable of acquiring an image of a real object and depth information indicating the distance between the terminal device 30 and the real object, similar to the input / output device 20.

[0123] 11, as indicated by reference symbol P31, the user brings the terminal device 30 into contact with or into proximity with a position in real space to be registered as a start point, and performs a predetermined operation related to registering the start point on the terminal device 30. In response to the operation, the system estimates the three-dimensional position in real space that the terminal device 30 has contacted or approached, based on the position and attitude of the terminal device 30 and the distance (depth) between the terminal device 30 and a real object 90, and registers the estimated position as a start point.

[0124] The registration of an end point is similar to that of a start point. For example, as shown by reference symbol P32, the user brings the terminal device 30 into contact with or into close proximity to a position in real space to be registered as an end point, and performs a predetermined operation for registering the end point on the terminal device 30. In response to the operation, the system estimates the three-dimensional position in real space that the terminal device 30 has contacted or approached based on the position and attitude of the terminal device 30 and the distance (depth) between the terminal device 30 and the real object 90, and registers the estimated position as the end point.

[0125] An example of a method for measuring a distance in real space using the information processing system according to this embodiment has been described above with reference to FIGS.

[0126] <4.2. Functional configuration> Next, an example of the functional configuration of the information processing system according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the functional configuration of the information processing system according to this embodiment. In this description, as shown in Fig. 11, the functional configuration of the information processing system according to this embodiment will be described with a focus on a case where a distance in real space is measured using a terminal device 30.

[0127] 12, the information processing system 2 according to the present embodiment includes, for example, a terminal device 30 and an information processing device 40. The terminal device 30 corresponds to the terminal device 30 shown in FIG.

[0128] First, the following description will focus on the configuration of the terminal device 30. As shown in FIG.

[0129] The imaging unit 301 is configured as a so-called stereo camera, and captures images of a subject from a plurality of different positions, i.e., an image of the subject from a viewpoint corresponding to the right eye side (i.e., a right eye image) and an image of the subject from a viewpoint corresponding to the left eye side (i.e., a left eye image).The imaging unit 301 then outputs the captured right eye image and left eye image to the information processing device 40.

[0130] The input unit 303 is configured with an input interface such as a touch panel, buttons, etc. When the input unit 303 receives an operation from a user, it outputs control information indicating the content of the operation to the information processing device 40. Note that the terminal device 30 may include a sound collection unit for collecting the user's voice as an input interface, similar to the input / output device 20 (see FIG. 4) according to the first embodiment described above.

[0131] The output unit 305 is configured with a display or the like, and displays display information such as images based on the control from the information processing device 40 .

[0132] The terminal device 30 may include a detection unit (not shown) for acquiring information indicating changes in the position and attitude of the terminal device 30, such as an acceleration sensor or an angular velocity sensor.

[0133] Next, a description will be given focusing on the configuration of the information processing device 40. As shown in Fig. 12, the information processing device 40 includes a depth detection unit 401, a position and orientation estimation unit 403, a trigger detection unit 405, a reference point estimation unit 407, a distance calculation unit 409, and an output control unit 411. Note that the depth detection unit 401, the position and orientation estimation unit 403, and the trigger detection unit 405 correspond to the depth detection unit 101, the position and orientation estimation unit 103, and the trigger detection unit 109 in the information processing device 10 (see Fig. 4) according to the first and second embodiments described above.

[0134] That is, the depth detection unit 401 acquires an image of a subject and depth information indicating a distance between the terminal device 30 and the subject, based on the right eye image and the left eye image captured by the imaging unit 301. Then, the depth detection unit 401 outputs the acquired image of the subject and the depth information to the position and orientation estimation unit 403. Note that the depth detection unit 401 may output the depth information to the reference point estimation unit 407.

[0135] The position and orientation estimation unit 403 is a configuration for executing a process related to estimation of the position and orientation of the terminal device 30 in the real space (so-called self-position estimation process). That is, the position and orientation estimation unit 403 acquires an image of a subject and depth information from the depth detection unit 401, and recognizes a real object captured as a subject in the image based on the acquired image and the acquired depth information. Furthermore, the position and orientation estimation unit 403 estimates the relative position and orientation of the terminal device 30 with respect to the real object based on the recognition result of the real object and the acquired depth information. At this time, the position and orientation estimation unit 403 may estimate the relative position and orientation of the terminal device 30 with respect to the recognized real object based on SLAM. Then, the position and orientation estimation unit 403 outputs information indicating the estimation result of the position and orientation of the terminal device 30 in the real space to the reference point estimation unit 407. Furthermore, the position and orientation estimation unit 403 may output, to the output control unit 411, information indicating the estimation result of the position and orientation of the terminal device 30 in the real space.

[0136] The trigger detection unit 405 acquires information indicating a user input via the input unit 303 from the input unit 303. Then, when the acquired information indicating the user input indicates a predetermined operation content, the trigger detection unit 405 uses the user input as a trigger to give an instruction associated with the operation content to the reference point estimation unit 407. Based on this configuration, for example, the trigger detection unit 405 uses the predetermined user input as a trigger to instruct the reference point estimation unit 407 to register reference points such as a start point and an end point related to distance measurement.

[0137] The reference point estimation unit 407 acquires information indicating an estimation result of the position and orientation of the terminal device 30 in real space from the position and orientation estimation unit 403. The reference point estimation unit 407 may also acquire depth information from the depth detection unit 401. Then, when receiving an instruction related to the registration of a reference point from the trigger detection unit 405, the reference point estimation unit 407 estimates the position and orientation of the terminal device 30. Based on the result, the positions in real space of reference points such as the start point and end point are estimated.

[0138] More specifically, when the reference point estimation unit 407 receives an instruction for registering a reference point from the trigger detection unit 405, it may regard the position of the specific part of the terminal device 30 at that time (in other words, the position of the terminal device 30) as the gaze point described in the first and second embodiments, and register the gaze point as a reference point. Furthermore, based on the estimation result of the position and attitude of the terminal device 30 and the acquired depth information, the reference point estimation unit 407 may regard a position (i.e., a position on a real object) that is separated from the position of the specific part of the terminal device 30 in the imaging direction of the imaging unit 301 by a distance indicated by the depth information as the gaze point, and register the gaze point as a reference point.

[0139] In this manner, the reference point estimation unit 407 registers reference points such as a start point and an end point based on instructions from the trigger detection unit 405, and outputs position information indicating the position of each of the registered reference points in real space to the distance calculation unit 409.

[0140] The distance calculation unit 409 corresponds to the distance calculation unit 113 in the information processing device 10 (see FIG. 4) according to the first and second embodiments described above. That is, the distance calculation unit 409 acquires position information indicating the position of each registered reference point in real space from the reference point estimation unit 407. Furthermore, the distance calculation unit 409 calculates the distance between a plurality of reference points based on the position information of each registered reference point, and outputs information indicating the calculation result of the distance to the output control unit 411. Furthermore, at this time, the distance calculation unit 409 may output information indicating the position information of each registered reference point to the output control unit 411. Furthermore, the distance calculation unit 409 may calculate the distance between a registered reference point (for example, a start point) and a gaze point, and output information indicating the calculated distance to the output control unit 411.

[0141] The output control unit 411 presents the virtual object to the user via the output unit 305 based on AR technology so that the virtual object is superimposed on real space.

[0142] As a specific example, the output control unit 411 acquires information indicating the recognition result of the real object and the estimation result of the position and orientation of the terminal device 30 in the real space from the position and orientation estimation unit 103. This enables the output control unit 411 to estimate the positional relationship between the terminal device 30 and the recognized real object in the real space. Then, the output control unit 411 may cause the output unit 211 to display an image in which a virtual object is superimposed on an image captured by the imaging unit 301 in accordance with the positional relationship between the terminal device 30 and the recognized real object in the real space.

[0143] Furthermore, the output control unit 411 may obtain information indicating the calculation result of the distance from the distance calculation unit 409, and display information corresponding to the calculation result of the distance on the basis of the information, on the output unit 305. This operation is similar to that of the output control unit 115 in the information processing device 10 according to the first and second embodiments described above.

[0144] 12 is merely an example, and the configuration of the information processing system 2 is not necessarily limited to the example shown in Fig. 12. In this respect, it is similar to the information processing system 1 according to the first embodiment described above. Also, as long as it is possible to acquire an image of an object in real space and depth information indicating the distance from the terminal device 30 to the object, the configuration for acquiring the image and the depth information is not particularly limited, which is also similar to the information processing system 1 according to the first embodiment described above.

[0145] An example of the functional configuration of the information processing system according to the present embodiment has been described above with reference to FIG.

[0146] <4.3. Processing> Next, an example of a series of process flows of the information processing system according to the present embodiment will be described with reference to Fig. 13, focusing in particular on the process of the information processing device 40. Fig. 13 is a flowchart showing an example of a series of process flows of the information processing system according to the present embodiment.

[0147] First, the information processing device 40 (depth detection unit 401) calculates the distance (depth) between the terminal device 30 and the captured subject (i.e., a real object) based on the right-eye image and the left-eye image captured by the imaging unit 301. As a result, an image of the real object and depth information indicating the distance between the terminal device 30 and the real object are acquired (S201).

[0148] Next, the information processing device 40 (position and orientation estimation unit 403) estimates the relative position and orientation of the terminal device 30 with respect to the real object captured in the image based on the acquired image and depth information. At this time, the position and orientation estimation unit 403 may also estimate the relative position and orientation of the terminal device 30 with respect to the recognized real object based on SLAM (S203).

[0149] Next, based on the estimation results of the position and attitude of the terminal device 30, the information processing device 40 (reference point estimation unit 407) regards the position of a specific part of the terminal device 30 as a gaze point, and estimates the three-dimensional position of the gaze point in real space as an absolute position (S205).

[0150] The information processing device 40 sequentially executes a series of processes indicated by reference signs S201 to S205 unless a predetermined trigger based on a user input via the input unit 303 is detected (S207, NO). Then, when a predetermined trigger based on a user input is detected (S207, YES), the information processing device 40 executes processes related to registration of the start point and the end point.

[0151] For example, when the start point is not registered (S209, NO), the information processing device 40 (reference point estimation unit 407) registers the three-dimensional position in real space of the gaze point when the trigger is detected as the start point (S211). After registering the start point, the information processing device 40 (distance calculation unit 409) calculates the distance from the registered start point to the current gaze point (S213).

[0152] Furthermore, when the start point has already been registered (S209, YES), the information processing device 40 (reference point estimation unit 407) registers the three-dimensional position in real space of the gaze point when the trigger is detected as the end point (S215). In this case, the information processing device 40 (distance calculation unit 409) calculates the distance from the registered start point to the end point. Then, the information processing device 40 (output control unit 411) causes the output unit 305 of the terminal device 30 to display information based on the calculation result of the distance (S217).

[0153] The information processing device 40 executes the above-mentioned series of processes until the user instructs the end of the process (for example, the end of the application) (S219, NO). Then, when the user instructs the end of the process (S219, YES), the information processing device 40 ends the above-mentioned series of processes.

[0154] An example of the flow of a series of processes in the information processing system according to this embodiment has been described above with reference to FIG. 13, focusing particularly on the processes in the information processing device 40.

[0155] <4.4. Evaluation> As described above, in the information processing system according to the present embodiment, the user, for example, touches or brings the terminal device 30 close to a desired position in real space, and The user specifies the start point and end point of the distance measurement in the real space by performing a predetermined operation on the terminal device 30. At this time, the information processing device 40 recognizes, for example, the estimated position of the terminal device 30 in the real space as the positions of reference points such as the start point and the end point. Furthermore, the information processing device 40 calculates (measures) the distance between the start point and the end point based on the recognition results of the positions of the start point and the end point in the real space. Then, the information processing device 40 presents display information based on the calculation results of the distance between the start point and the end point to the user via the terminal device 30. With this configuration, the user can measure the distance in the real space by a simpler operation using the terminal device 30 without actually using a tool such as a ruler or a tape measure to measure the distance.

[0156] In particular, with the information processing system of this embodiment, in a situation where a user is able to touch or approach a real object to be measured for distance, it becomes possible to measure distance with a simpler procedure than with the information processing systems of the first and second embodiments described above.

[0157] <<5. Fourth embodiment>> Next, an example of an information processing system according to a fourth embodiment of the present disclosure will be described.

[0158] <5.1.Measurement method> First, an example of a method for measuring distance in real space using the information processing system according to this embodiment will be described with reference to Fig. 14. For example, Fig. 14 is an explanatory diagram for explaining an overview of the information processing system according to this embodiment, and shows an example of a method for measuring distance in real space. Note that Fig. 14 presents both real objects and virtual objects in order to make the features of the information processing system according to this embodiment easier to understand.

[0159] The information processing system according to this embodiment recognizes an object such as the terminal device 30 based on an image captured by an imaging unit (the first imaging unit 201 configured as a stereo camera described above) provided in the input / output device 20. Then, the system presents a virtual object V43 for specifying a reference point such as a start point or an end point to the recognized object (i.e., the terminal device 30) via the input / output device 20 to the user in accordance with the recognition result, so that the virtual object V43 is superimposed on the recognized object (i.e., the terminal device 30). For example, in the example shown in FIG. 14, the system recognizes a marker V41 presented on a display or the like of the terminal device 30, and recognizes the terminal device 30 in accordance with the recognition result of the marker V41. This makes it possible for the system to recognize the position and attitude of the terminal device 30 in real space based on the recognition result of the marker V41. Also, the system presents the virtual object V43 in accordance with the recognition result of the terminal device 30 so as to extend from the terminal device 30 in the direction in which the terminal device 30 is facing. While checking the virtual object V43 via the input / output device 20, the user changes the position and orientation of the terminal device 30 to specify positions in real space to be registered as the start and end points of distance measurement in real space using the virtual object V43.

[0160] Specifically, first, the user, wearing the input / output device 20, visually recognizes the terminal device 30 on which the marker V41 is presented. In response to this operation, the system recognizes the position and orientation of the terminal device 30 on which the marker V41 is presented, based on an image captured by an imaging unit of the input / output device 20, and presents a virtual object V43 to the user via the input / output device 20 based on the recognition result. At this time, the system may control the display mode (e.g., length, etc.) of the virtual object V43 in response to a predetermined operation on the terminal device 30 or the input / output device 20.

[0161] Next, the user adjusts the position and orientation of the terminal device 30 as shown by reference numeral P41. By this, the user indicates a position in real space to be registered as a starting point by the presented virtual object V43, and performs a predetermined operation related to registering the starting point on the terminal device 30 or the input / output device 20. In response to this operation, the system registers the position in real space designated by the virtual object V43 as the starting point. For example, in the example shown in FIG. 14, the system registers the position in real space indicated by the arrow-shaped virtual object V43 as the starting point.

[0162] The user also registers the end point in the same manner as the start point. For example, as shown by reference symbol P42, the user uses the presented virtual object V43 to point to a position in real space to be registered as the end point, and performs a predetermined operation related to registering the end point on the terminal device 30 or the input / output device 20. In response to this operation, the system registers the position in real space pointed to by the arrow-shaped virtual object V43 as the end point.

[0163] The operation after the start point and the end point are registered is the same as that of the first to third embodiments. That is, the system measures (calculates) the distance between the registered start point and the end point based on the three-dimensional positions of the start point and the end point, and presents various display information to the user via the input / output device 20 according to the measurement result. With this configuration, the user can measure the distance in real space by a simpler operation using the input / output device 20 without actually using a tool such as a ruler or a tape measure to measure the distance.

[0164] 14, an example has been described in which the terminal device 30 is used as an object on which the virtual object V43 is superimposed. On the other hand, if the user can specify a reference point such as a start point or an end point by using the presented virtual object V43, the object on which the virtual object V43 is superimposed (in other words, the object recognized to present the virtual object V43) is not necessarily limited to the terminal device 30. For example, instead of the terminal device 30, a card on which the marker V41 is printed may be used.

[0165] An example of a method for measuring a distance in real space using the information processing system according to this embodiment has been described above with reference to FIG.

[0166] <5.2. Functional configuration> Next, an example of the functional configuration of the information processing system according to this embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram showing an example of the functional configuration of the information processing system according to this embodiment.

[0167] 15, the information processing system 3 according to the present embodiment includes, for example, an input / output device 20, a terminal device 30, and an information processing device 50. The input / output device 20 and the terminal device 30 correspond to the input / output device 20 and the terminal device 30 shown in FIG.

[0168] First, the description will be made with a focus on the configuration of the input / output device 20. As shown in Fig. 15, the input / output device 20 according to this embodiment includes a first imaging unit 201, an input unit 205, and an output unit 211. Note that the first imaging unit 201, the input unit 205, and the output unit 211 are substantially similar to the first imaging unit 201, the input unit 205, and the output unit 211 in the input / output device 20 according to the first embodiment described above (see Fig. 4), and detailed description thereof will be omitted.

[0169] Next, the description will be focused on the configuration of the terminal device 30. As shown in Fig. 15, the terminal device 30 includes an input unit 303. Note that the input unit 303 is substantially similar to the input unit 303 in the terminal device 30 according to the third embodiment (see Fig. 12) described above, and detailed description thereof will be omitted.

[0170] Next, the configuration of the information processing device 50 will be described. As shown in FIG. The device 50 includes a depth detection unit 501, a position and orientation estimation unit 503, an object recognition unit 505, a trigger detection unit 507, a reference point estimation unit 509, a distance calculation unit 511, and an output control unit 513. Note that the depth detection unit 501 and the position and orientation estimation unit 503 correspond to the depth detection unit 501 and the position and orientation estimation unit 503 in the information processing device 10 (see FIG. 4) according to the first and second embodiments described above.

[0171] That is, the depth detection unit 501 acquires an image of a subject and depth information indicating a distance between the input / output device 20 and the subject, based on the right eye image and the left eye image captured by the first imaging unit 201. Then, the depth detection unit 501 outputs the acquired image of the subject and the depth information to the position and orientation estimation unit 503.

[0172] The position and orientation estimation unit 503 is a configuration for executing a process related to estimation of the position and orientation of the input / output device 20 in the real space (so-called self-position estimation process). That is, the position and orientation estimation unit 503 acquires an image of a subject and depth information from the depth detection unit 501, and recognizes a real object captured as a subject in the image based on the acquired image and the acquired depth information. Then, the position and orientation estimation unit 503 estimates the relative position and orientation of the input / output device 20 with respect to the real object based on the recognition result of the real object and the acquired depth information. At this time, the position and orientation estimation unit 503 may estimate the relative position and orientation of the input / output device 20 with respect to the recognized real object based on SLAM. Then, the position and orientation estimation unit 503 outputs information indicating the estimation result of the position and orientation of the input / output device 20 in the real space to the reference point estimation unit 509. Furthermore, the position and orientation estimation unit 503 may output, to the output control unit 513, information indicating the estimation result of the position and orientation of the input / output device 20 in the real space.

[0173] The object recognition unit 505 is configured to recognize the position and orientation of the terminal device 30 based on the image captured by the first imaging unit 201. For example, the object recognition unit 505 may recognize a marker V41 presented on the terminal device 30 as shown in FIG. 14 from the image captured by the first imaging unit 201, and recognize the position and orientation of the terminal device 30 based on the recognition result. Note that the method of recognizing the terminal device 30 is substantially similar to the method of recognizing a real object captured as a subject in an image by the position and orientation estimation unit 503, so detailed description will be omitted. Then, the object recognition unit 505 outputs information indicating the recognition result of the position and orientation of the terminal device 30 to the reference point estimation unit 509. The object recognition unit 505 may also output information indicating the recognition result of the position and orientation of the terminal device 30 to the output control unit 513.

[0174] As described above, the processing of the object recognition unit 505 may be similar to at least a part of the processing of the position and orientation estimation unit 503. Therefore, the position and orientation estimation unit 503 may execute the processing of the object recognition unit 505.

[0175] The trigger detection unit 507 acquires information indicating a user input via the input unit 303 from the input unit 303. The trigger detection unit 507 may also acquire information indicating a user input via the input unit 205 from the input unit 205. Then, when the acquired information indicating the user input indicates a predetermined operation content, the trigger detection unit 507 uses the user input as a trigger to give an instruction associated with the operation content to the reference point estimation unit 509. Based on this configuration, for example, the trigger detection unit 507 uses the predetermined user input as a trigger to instruct the reference point estimation unit 509 to register reference points such as a start point and an end point related to distance measurement.

[0176] The reference point estimation unit 509 acquires information indicating the estimation result of the position and orientation of the input / output device 20 in the real space from the position and orientation estimation unit 503. The reference point estimation unit 509 also acquires information indicating the recognition result of the position and orientation of the terminal device 30 from the object recognition unit 505. Then, when the reference point estimation unit 509 receives an instruction related to the registration of a reference point from the trigger detection unit 507, Next, based on the estimation results of the position and orientation of the input / output device 20 and the recognition results of the position and orientation of the terminal device 30, the positions in real space of reference points such as the start point and end point are estimated.

[0177] More specifically, based on the recognition result of the position and orientation of the terminal device 30, the reference point estimation unit 509 recognizes the position and orientation of the terminal device 30 in the real space as a relative position based on the position and orientation of the input / output device 20. Furthermore, the reference point estimation unit 509 recognizes the position and orientation of the input / output device 20 in the real space based on the information acquired from the position and orientation estimation unit 503. This enables the reference point estimation unit 509 to estimate the three-dimensional position and orientation of the terminal device 30 in the real space as, for example, an absolute position.

[0178] Next, the reference point estimation unit 509 estimates the position in real space to which the terminal device 30 is directed (i.e., the position in real space specified by the virtual object V41 in FIG. 14) based on the estimation result of the three-dimensional position and orientation of the terminal device 30 in real space. Note that the position in real space estimated by such processing corresponds to the gaze point described in the first and second embodiments. Therefore, in this description, for convenience, the position in real space estimated by the above processing (i.e., the position in real space to which the terminal device 30 is directed) is referred to as the "gazing point."

[0179] At this time, the reference point estimation unit 509 may estimate a three-dimensional position of the gaze point in the real space according to the operation result on the virtual object V41. More specifically, the reference point estimation unit 509 may estimate, as the position of the gaze point, a position that is separated from the position of the terminal device 30 in the real space by the length of the virtual object V41 in the direction in which the terminal device 30 is facing according to the adjustment result of the length of the virtual object V41.

[0180] Then, the reference point estimation unit 509 registers reference points such as a start point and an end point based on instructions from the trigger detection unit 507 regarding the registration of reference points, and outputs position information indicating the position of each of the registered reference points in real space to the distance calculation unit 511.

[0181] Furthermore, the reference point estimation unit 509 may sequentially output information indicating the position of the gaze point in real space to the distance calculation unit 511. This enables the distance calculation unit 511 to recognize in real time the position in real space to which the terminal device 30 is directed.

[0182] The process of the distance calculation unit 511 is substantially the same as that of the distance calculation unit 113 in the information processing device 10 according to the first and second embodiments described above. That is, the distance calculation unit 511 acquires position information indicating the position of each registered reference point in real space from the reference point estimation unit 509. Furthermore, the distance calculation unit 511 calculates the distance between a plurality of reference points based on the position information of each registered reference point, and outputs information indicating the calculation result of the distance to the output control unit 513. Furthermore, at this time, the distance calculation unit 511 may output information indicating the position information of each registered reference point to the output control unit 513.

[0183] Furthermore, the distance calculation unit 511 may sequentially acquire information indicating the position of the gaze point in real space from the reference point estimation unit 509. In this case, the distance calculation unit 511 may, for example, calculate the distance between a registered reference point (for example, a start point) and the gaze point, and output information indicating the calculation result of the distance to the output control unit 513. At this time, the distance calculation unit 511 may also output information indicating the position of the gaze point in real space to the output control unit 513.

[0184] The output control unit 513 presents the virtual object to the user via the output unit 211 so that the virtual object is superimposed on the real space based on the AR technology. Note that the process related to the presentation of the virtual object by the output control unit 513 is substantially similar to that of the output control unit 115 of the information processing device 10 (see FIG. 4) according to the first and second embodiments described above, and therefore will not be described in detail. Detailed explanation will be omitted.

[0185] For example, the output control unit 513 acquires information indicating the recognition result of the position and orientation of the terminal device 30 from the object recognition unit 505. The output control unit 513 also acquires information indicating the estimation result of the position and orientation of the input / output device 20 in the real space from the position and orientation estimation unit 503. Then, the output control unit 513 may estimate the three-dimensional position and orientation of the terminal device 30 in the real space based on the acquired information indicating the recognition result of the position and orientation of the terminal device 30 and the information indicating the estimation result of the position and orientation of the input / output device 20 in the real space. Then, for example, based on the estimation result, the output control unit 513 may display the virtual object V41 on the output unit 211 so that the virtual object V41 is superimposed on the terminal device 30 as shown in FIG. 14.

[0186] Furthermore, the output control unit 513 may obtain information indicating the calculation result of the distance from the distance calculation unit 511, and display information corresponding to the calculation result of the distance on the basis of the information on the output unit 211. This operation is similar to that of the output control unit 115 in the information processing device 10 according to the first and second embodiments described above.

[0187] 15 is merely an example, and the configuration of the information processing system 3 is not necessarily limited to the example shown in FIG 15. In this respect, it is similar to the information processing system 1 according to the first embodiment described above. In addition, as long as it is possible to acquire an image of an object in real space and depth information indicating the distance from the input / output device 20 to the object, the configuration for acquiring the image and the depth information is not particularly limited, which is also similar to the information processing system 1 according to the first embodiment described above.

[0188] An example of the functional configuration of the information processing system according to the present embodiment has been described above with reference to FIG.

[0189] 5.3. Processing Next, an example of a series of process flows of the information processing system according to the present embodiment will be described with reference to Fig. 16, focusing in particular on the process of the information processing device 50. Fig. 16 is a flowchart showing an example of a series of process flows of the information processing system according to the present embodiment.

[0190] First, the information processing device 50 (depth detection unit 501) calculates the distance (depth) between the input / output device 20 and the captured subject (i.e., a real object) based on the right-eye image and the left-eye image captured by the first imaging unit 201. As a result, an image of the real object and depth information indicating the distance between the input / output device 20 and the real object are acquired (S301).

[0191] Next, the information processing device 50 (position and orientation estimation unit 503) estimates the relative position and orientation of the input / output device 20 with respect to the real object captured in the image based on the acquired image and depth information. At this time, the position and orientation estimation unit 503 may also estimate the relative position and orientation of the input / output device 20 with respect to the recognized real object based on SLAM (S303).

[0192] Furthermore, the information processing device 50 (object recognition unit 505) recognizes the position and orientation of the terminal device 30 based on the right eye image and the left eye image captured by the first imaging unit 201. Furthermore, the information processing device 50 (output control unit 513) presents the virtual object V41 to the user via the input / output device 20 so that the virtual object V41 is superimposed on the terminal device 30 based on the recognition result of the position and orientation of the terminal device 30 (S305). Note that the method of recognizing the position and orientation of the terminal device 30 is as described above as the processing of the object recognition unit 505. be.

[0193] Next, the information processing device 50 (reference point estimation unit 509) estimates the three-dimensional position in real space toward which the terminal device 30 is directed (i.e., the position of the gaze point in real space) based on the estimation result of the position and attitude of the input / output device 20 and the recognition result of the position and attitude of the terminal device 30. Note that at this time, the information processing device 50 may estimate the three-dimensional position of the gaze point in real space according to the operation result on the virtual object V41 presented according to the recognition result of the position and attitude of the terminal device 30 (S307). Note that the method of estimating the position of the gaze point is as described above as the processing of the reference point estimation unit 509.

[0194] The information processing device 50 sequentially executes a series of processes indicated by reference signs S301 to S307 unless a predetermined trigger based on a user input via the input unit 205 or the input unit 303 is detected (S309, NO). Then, when a predetermined trigger based on a user input is detected (S309, YES), the information processing device 50 executes processes related to registration of the start point and the end point.

[0195] For example, when the start point is not registered (S311, NO), the information processing device 50 (reference point estimation unit 509) registers the three-dimensional position in real space of the gaze point when the trigger is detected as the start point (S313). After registering the start point, the information processing device 50 (distance calculation unit 511) calculates the distance from the registered start point to the current gaze point (S315).

[0196] Furthermore, when the start point has already been registered (S311, YES), the information processing device 50 (reference point estimation unit 509) registers the three-dimensional position in real space of the gaze point when the trigger is detected as the end point (S317). In this case, the information processing device 50 (distance calculation unit 511) calculates the distance from the registered start point to the end point. Then, the information processing device 50 (output control unit 513) causes the output unit 211 of the input / output device 20 to display information based on the calculation result of the distance.

[0197] The information processing device 50 executes the above-mentioned series of processes until the user instructs the end of the process (for example, the end of the application) (S321, NO). Then, when the user instructs the end of the process (S321, YES), the information processing device 50 ends the above-mentioned series of processes.

[0198] An example of the flow of a series of processes in the information processing system according to this embodiment has been described above with reference to FIG. 16, focusing particularly on the processes in the information processing device 50.

[0199] <5.4.Evaluation> As described above, in the information processing system according to the present embodiment, the information processing device 50 recognizes an object such as the terminal device 30 based on an image captured by an imaging unit provided in the input / output device 20. Then, the information processing device 50 presents the virtual object V43 for specifying a reference point such as a start point or an end point to the user via the input / output device 20 in such a manner that the virtual object V43 is superimposed on the recognized object according to the recognition result. With such a configuration, the user specifies positions in real space to be registered as the start point and the end point by the virtual object V43 by changing the position and orientation of the terminal device 30 while checking the virtual object V43 via the input / output device 20. At this time, the information processing device 50, for example, estimates the position in real space specified by the virtual object V43 based on the recognition result of the position and orientation of the terminal device 30, and registers the estimated position as the position of a reference point such as a start point or an end point.

[0200] Furthermore, the information processing device 50 calculates (measures) the distance between the start point and the end point based on the registration results of the positions of the start point and the end point in real space. Then, the information processing device 50 presents display information based on the calculation results of the distance between the start point and the end point to the user via the input / output device 20. With this configuration, the user can measure the distance in real space by simpler operations using the input / output device 20 and the terminal device 30, without actually measuring the distance using tools such as a ruler or tape measure.

[0201] <<6. Modifications>> Next, a modified example of the information processing system according to an embodiment of the present disclosure will be described.

[0202] <6.1. Variation 1: Measuring the length of a curve> First, an information processing system according to Modification 1 will be described. In each of the above-described embodiments, an example has been described in which a reference point such as a start point or an end point is specified to measure the distance between a plurality of reference points (i.e., a straight-line distance). On the other hand, the information processing system according to Modification 1 is configured to be capable of measuring not only a straight-line distance but also a curved length such as a distance on a curved surface. For example, FIG. 17 is an explanatory diagram for explaining an overview of the information processing system according to Modification 1. Note that in FIG. 17, both a real object and a virtual object are presented together in order to make the characteristics of the information processing system according to Modification 1 easier to understand.

[0203] Specifically, in the information processing system according to the first modification, the user specifies, via the input / output device 20 or the terminal device 30, a route (trajectory) for which the distance is to be measured.

[0204] For example, in the example shown in FIG. 17, the user places a virtual object V51 on the surface of a real object 91 by operating the terminal device 30, as in the case of measuring a distance by wrapping a so-called tape measure around an object, to set a path (trajectory) for which distance is to be measured in real space. At this time, the system determines whether or not a part V51a of the virtual object V51 that has not yet been placed has come into contact with the real object 91, for example, according to the positional relationship between the terminal device 30 and the real object 91. Note that the system may recognize the positional relationship between the virtual object V51 and the real object 91 in real space according to, for example, an estimation result of the position and orientation of the terminal device 30 based on a self-location estimation technology such as SLAM. Then, the system may recognize the part of the part V51a of the virtual object V51 that is determined to have come into contact with the real object 91 as the part V51b set as the path for which distance is to be measured.

[0205] As another example, the user may set, for example, a route to be the object of distance measurement in the same manner as when specifying a position in real space in each of the above-mentioned embodiments. As a more specific example, the user may move his / her line of sight while wearing the input / output device 20, and trace a route (trajectory) in real space with the position (i.e., the gaze point) to which the line of sight is directed, thereby specifying the route as the object of distance measurement. In this case, for example, the system may detect a change in the user's line of sight, and recognize the trajectory according to the change in the gaze point to which the line of sight is directed, as the route to be the object of distance measurement.

[0206] In this manner, the system recognizes the route (trajectory) specified by the user via the input / output device 20 or the terminal device 30, and measures the distance of the route. With this configuration, the information processing system according to the first modification can measure not only the straight-line distance, but also the distance along a curved trajectory. Note that a commonly known method can be applied to the method of measuring the length of the curve, so a detailed description will be omitted.

[0207] The information processing system according to the first modification has been described above with reference to FIG.

[0208] <6.2. Modification 2: An example of a measurement method based on the operation of a virtual object> Next, an information processing system according to Modification 2 will be described. In the above-mentioned embodiments and modifications, the distance between a plurality of reference points specified via the input / output device 20 or the terminal device 30, or the distance along a specified trajectory, is measured. In contrast, in Modification 2, a procedure for measuring the distance in real space based on a procedure different from that of the above-mentioned embodiments and modifications, and an example of control for realizing the procedure will be described. For example, FIG. 18 is an explanatory diagram for explaining the outline of the information processing system according to Modification 2. In FIG. 18, both real objects and virtual objects are presented together to make the characteristics of the information processing system according to Modification 2 easier to understand.

[0209] Specifically, in the information processing system according to variant example 2, the user measures distances in real space by adjusting the size of a virtual object presented by the system to be superimposed on real space through operations via the input / output device 20 or the terminal device 30.

[0210] For example, Fig. 18 shows an example of measuring the distance between a real object 93a and a real object 93b. Specifically, in the example shown in Fig. 18, the system receives a predetermined operation from the user and presents a rectangular parallelepiped virtual object V61 to the user via the output unit 305 of the terminal device 30 so that the virtual object V61 is superimposed on a desired position in the real space.

[0211] The system also presents display information V63 indicating the size (length) of a predetermined portion of the presented virtual object V61 in association with the virtual object V61. For example, in the example shown in Fig. 18, the system presents display information V64a, V63b, and V63c indicating the length, width, and depth of the rectangular parallelepiped virtual object V61. Note that the system may recognize the position in the real space where the virtual object V61 is superimposed and the size of the virtual object V61 in the real space according to the estimation results of the position and attitude of the terminal device 30 based on a self-position estimation technology such as SLAM.

[0212] With this configuration, the user can measure the distance by adjusting the position in the real space where the presented virtual object V61 is superimposed and the size of each part of the virtual object V61 (for example, the vertical, horizontal, and depth widths) by operating the input unit 303 (for example, a touch panel, etc.) of the terminal device 30. For example, in the example shown in FIG. 18, the user adjusts the position and width of the virtual object V61 so that the width of the virtual object V61 fits exactly between the real objects V93a and V93b. With this procedure, the user can measure the distance between the real objects 93a and 93b as the width of the virtual object V61 and recognize the measurement result from the display information V63b.

[0213] As another example, the user may adjust the size of each part of the virtual object V61 so that the target real object fits inside the virtual object V61 in a rectangular parallelepiped shape. By performing such an operation, it is possible to measure the size of a box for storing an object (i.e., a real object) that is not rectangular, such as a stuffed toy or luggage.

[0214] The information processing system according to the second modification has been described above with reference to FIG.

[0215] <6.3. Modification 3: An example of linking multiple devices> Next, an information processing system according to Modification 3 will be described. In Modification 3, an example of the configuration and control in the case where a plurality of devices such as the input / output device 20 and the terminal device 30 are linked together to measure the distance between a plurality of positions farther apart in the real space will be described. For example, FIG. 19 is an explanatory diagram for explaining the outline of the information processing system according to Modification 3. In Fig. 19, in order to make the characteristics of the information processing system according to the third modification easier to understand, both real objects and virtual objects are presented.

[0216] 19, the input / output devices 20a and 20b operate in cooperation with the information processing device 10. The information processing device 10 is configured to be capable of transmitting and receiving information to and from each of the input / output devices 20a and 20b via a predetermined communication path (e.g., a wireless communication path, etc.). With such a configuration, for example, a start point of distance measurement is specified by an operation via the input / output device 20a, and an end point is specified by an operation via the input / output device 20b.

[0217] At this time, the information processing device 10 recognizes the positional relationship between the input / output devices 20a and 20b, for example, based on the result of self-position estimation by each of the input / output devices 20a and 20b, or the result of positioning the respective positions of the input / output devices 20a and 20b by a positioning system such as a GPS (Global Positioning System). This enables the information processing device 10 to recognize the positional relationship in real space between a start point specified via the input / output device 20a and an end point specified via the input / output device 20b, based on the recognition result of the positional relationship between the input / output devices 20a and 20b.

[0218] Then, the information processing device 10 may measure the distance between the start point and the end point according to the recognition result of the positional relationship between the start point and the end point in real space. At this time, the information processing device 10 may present display information according to the distance measurement result to the user via the input / output device 20a or the input / output device 20b.

[0219] 19 is merely an example, and the configuration of the information processing system according to Modification 3 is not necessarily limited to the example shown in FIG 19 as long as the above-mentioned functions can be realized. As a specific example, either of the input / output devices 20a and 20b may perform the function corresponding to the information processing device 10. As another example, the input / output devices 20a and 20b may transmit and receive information to each other, thereby recognizing their positional relationship in real space, and each of them may independently perform processing related to distance measurement.

[0220] The information processing system according to the third modification has been described above with reference to FIG.

[0221] <<7. Application Examples>> Next, application examples of the information processing system according to each embodiment of the present disclosure will be described.

[0222] (Selective switching of measurement methods) As described above, the measurement methods described as the embodiments and modifications use different devices and procedures, and therefore the system may be configured to selectively use at least a portion of the above-described measurement methods depending on the situation.

[0223] As a more specific example, the measurement method described as the first embodiment (see FIG. 2) is more suitable for a situation in which the size (distance, etc.) of an object is measured when the user is apart from the object. The measurement method described as the second embodiment (see FIG. 6) is more suitable for a situation in which the user can approach the object. The measurement method described as the third embodiment (see FIGS. 10 and 11) is more suitable for a situation in which the user can bring the input / output device 20 or the terminal device 30 into contact with or close to the object. Therefore, assuming such differences in situations, the system may be configured to set a mode corresponding to each measurement method (e.g., "long distance mode", "short distance mode", etc.) and selectively execute each of the above-mentioned measurement methods according to the mode designated by the user.

[0224] Also, a mode for selectively executing each measurement method may be set according to the difference in the procedure of each measurement method. As a specific example, the measurement method described as the first embodiment (see FIG. 2) is capable of measuring the distance based on a simpler procedure, but is capable of registering only a position on the surface of a real object as a reference point. In contrast, the measurement method described as the second embodiment (see FIG. 6) requires more procedures than the first embodiment, but is capable of registering a position in a real space where no real object exists as a reference point. Therefore, the system may present, for example, the measurement method described as the first embodiment as a "simple mode" and present other measurement methods (for example, the measurement method described as the second embodiment) as a "detailed mode."

[0225] (An example of how to specify a reference point) In addition, the method of setting reference points such as the start point and the end point is not limited to the examples shown in each measurement procedure described above, and may be appropriately changed depending on the situation. As a specific example, the reference points may be automatically registered by using so-called object recognition based on image analysis, such as face recognition, hand recognition, and foot recognition.

[0226] As a more specific example, the information processing system according to an embodiment of the present disclosure may be used to measure the jumping distance in the long jump. In this case, the system, for example, captures an image of an athlete jumping with the imaging unit of the input / output device 20, and performs image analysis on the image to recognize the jumping and landing movements of the athlete. When the system recognizes the landing of the athlete, it recognizes the feet of the athlete at the time of landing and recognizes the position of the feet as the landing point. This makes it possible for the system to measure the jumping distance in the long jump by measuring the distance from the position of the takeoff board as the starting point to the recognized landing point as the end point.

[0227] (Displaying various information using distance measurement results) In addition, the measurement results of the distance in the real space may be used to present various kinds of information. For example, Fig. 20 and Fig. 21 are explanatory diagrams for explaining an example of a method of presenting various kinds of information using the measurement results of the distance in the real space, and show an example of a case where an image of setting up a tent or the like is presented according to the measurement results of the distance during camping.

[0228] Specifically, first, as shown in Fig. 20, the user uses the input / output device 20 to measure the dimensions of the area in which a tent or the like is to be placed based on the various measurement methods described above. At this time, the system estimates the position and orientation of the input / output device 20 in the real space based on a self-location estimation technique such as SLAM, and recognizes the position and dimensions of the area specified by the user in the real space based on the estimation result.

[0229] The system also stores information indicating the shape and size of an object to be installed, such as a tent, in advance. Based on the information indicating the shape and size of the object and the recognition result of the position and dimensions of the area specified by the user in the real space, the system presents virtual objects V71 to V73 imitating the object, such as a tent, so as to be superimposed on the area in the real space at the size in the real space. Based on such a configuration, the system presents an installation image of an object, such as a tent, in the area specified by the user as virtual objects V71 to V73 superimposed on the real space, as shown in FIG. 21, for example. This allows the user to recognize a more suitable position in the real space for driving in a tent stake, based on the installation image as shown in FIG. 21, for example.

[0230] In addition, the scene in which the mechanism described above with reference to Figs. 20 and 21 is used is not limited to the above-mentioned example of use during camping. For example, by using the measurement results of the dimensions of a room, it is possible to present an image of how furniture should be installed. By using the measurement results of the inside dimensions of a suitcase, it is possible to present an image of what will be stored in the suitcase. As another example, by using the measurement results of the sizes of parts of the body such as the arms and legs, an image of what the suitcase will look like when worn with clothes may be presented.

[0231] (Application to area and volume measurements) In addition, although the above describes the measurement of one-dimensional sizes such as distance, it is also possible to measure two-dimensional sizes such as area, and three-dimensional sizes such as volume.

[0232] For example, Fig. 22 is a diagram showing an example of a measurement procedure for measuring an area. In the example shown in Fig. 22, the system registers three or more reference points based on a user's specification via the input / output device 20 or the terminal device 30, and measures the area of ​​a region surrounded by the three or more reference points.

[0233] Specifically, the user sequentially registers reference points indicated by reference symbols V81 to V83 based on an operation via the input / output device 20. At this time, the system recognizes the positions of the registered reference points V81 to V83 in real space, for example, based on the estimation results of the position and orientation of the input / output device 20 in real space. Then, the system may calculate the area of ​​the region surrounded by the reference points V81 to V83 based on the positions of the recognized reference points V81 to V83 in the real section. In addition, at this time, the system may present display information indicating the calculation results of the area to the user via the input / output device 20.

[0234] 23 is a diagram showing an example of a measurement procedure for measuring a volume. For example, the system registers four or more reference points based on a user's specification via the input / output device 20 or the terminal device 30, and measures the volume of a three-dimensional area formed based on the sides connecting the four or more reference points.

[0235] Specifically, FIG. 23 shows an example of measuring the volume of a real object having a rectangular parallelepiped shape. In the example shown in FIG. 23, the user registers the three sides of the real object, namely, the length, width, and depth, by sequentially specifying the reference points indicated by reference signs V84 to V88 based on an operation via the input / output device 20. At this time, the system recognizes the positions of the registered reference points V84 to V88 in the real space based on, for example, the estimation results of the position and orientation of the input / output device 20 in the real space. Furthermore, the system calculates the length, width, and depth of the real object to be measured based on the recognition results of the positions of the reference points V84 to V88. Then, the system may calculate the volume of the real object based on the calculation results of the length, width, and depth of the real object to be measured. Furthermore, at this time, the system may present display information indicating the calculation results of the length, width, and depth of the real object to be measured and the calculation result of the volume to the user via the input / output device 20.

[0236] The application examples of the information processing system according to each embodiment of the present disclosure have been described above. Note that the above-mentioned examples are merely examples, and do not limit the application targets of the information processing system according to each embodiment of the present disclosure. As a specific example, the information processing system according to each embodiment of the present disclosure may be used, for example, for measuring length at a work site or the like.

[0237] <<8. Hardware configuration example>> Next, an example of a hardware configuration of so-called information processing devices such as the information processing devices 10, 40, and 50, the input / output device 20, and the terminal device 30 according to an embodiment of the present disclosure will be described with reference to Fig. 24. Note that this description focuses on the information processing device 10. For example, Fig. 24 shows an example of a hardware configuration of the information processing device 10 according to an embodiment of the present disclosure. FIG.

[0238] As shown in FIG. 24, the information processing device 10 includes a central processing unit (CPU) 901, a read only memory (ROM) 903, and a random access memory (RAM) 905. The information processing device 10 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Furthermore, the information processing device 10 may include an imaging device 933 and a sensor 935 as necessary. The information processing device 10 may have a processing circuit such as a digital signal processor (DSP) or an application specific integrated circuit (ASIC) instead of or together with the CPU 901.

[0239] The CPU 901 functions as an arithmetic processing device and a control device, and controls all or part of the operations in the information processing device 10 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and arithmetic parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. The CPU 901, the ROM 903, and the RAM 905 are connected to each other by a host bus 907 constituted by an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909. The aforementioned depth detection unit 101, position and orientation estimation unit 103, gaze detection unit 105, gaze point detection unit 107, trigger detection unit 109, reference point estimation unit 111, distance calculation unit 113, and output control unit 115, etc., can be realized, for example, by a CPU 901.

[0240] The input device 915 is a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. The input device 915 may include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device using infrared rays or other radio waves, or an external connection device 929 such as a mobile phone that supports the operation of the information processing device 10. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the CPU 901. The user inputs various data to the information processing device 10 or instructs the information processing device 10 to perform processing operations by operating the input device 915. For example, the operation unit 207 and the sound collection unit 209 described above can be realized by the input device 915. In addition, the imaging device 933 described later can also function as an input device by imaging the movements of the user's hands and fingers.

[0241] The output device 917 is configured with a device capable of visually or audibly notifying the user of the acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro-Luminescence) display, a projector, a hologram display device, an audio output device such as a speaker and a headphone, and a printer device. The output device 917 outputs the result obtained by the processing of the information processing device 10 as a video such as text or an image, or outputs audio or sound. The output device 917 may also include a light for brightening the surroundings. The output unit 211 described above may be realized by the output device 917, for example.

[0242] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 10. The storage device 919 is configured, for example, with a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, and various data acquired from the outside.

[0243] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built into or externally attached to the information processing device 10. The drive 921 reads out information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. The drive 921 also writes information onto the attached removable recording medium 927.

[0244] The connection port 923 is a port for directly connecting a device to the information processing device 10. The connection port 923 may be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. The connection port 923 may also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. By connecting an external connection device 929 to the connection port 923, various types of data may be exchanged between the information processing device 10 and the external connection device 929.

[0245] The communication device 925 is, for example, a communication interface configured with a communication device for connecting to the communication network 931. The communication device 925 may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 925 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The communication network 931 connected to the communication device 925 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0246] The imaging device 933 is a device that captures real space using various members such as an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and a lens for controlling the formation of a subject image on the imaging element, and generates a captured image. The imaging device 933 may capture a still image or a moving image. The first imaging unit 201 and the second imaging unit 203 described above may be realized by the imaging device 933, for example.

[0247] The sensor 935 is, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, a light sensor, a sound sensor, etc. The sensor 935 acquires information on the state of the information processing device 10 itself, for example, the attitude of the housing of the information processing device 10, and information on the surrounding environment of the information processing device 10, for example, the brightness and noise around the information processing device 10. The sensor 935 may also include a Global Positioning System (GPS) sensor that receives a GPS signal and measures the latitude, longitude, and altitude of the device.

[0248] An example of the hardware configuration of the information processing device 10 has been described above. The present invention may be configured using conventional components, or may be configured using hardware specialized for the functions of each component. Such a configuration may be appropriately changed depending on the technical level at the time of implementation.

[0249] It is also possible to create a program for causing hardware such as a processor, memory, and storage built into a computer to perform functions equivalent to those of the configuration of the information processing device 10. Also, a computer-readable storage medium on which the program is recorded may be provided.

[0250] <<9. Conclusion>> Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0251] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0252] Note that the following configurations also fall within the technical scope of the present disclosure. (1) an acquisition unit that acquires an image captured by a predetermined imaging unit and position information based on at least one of a position and an orientation of the imaging unit; an estimation unit that estimates a first position and a second position in a real space based on a first image and a second image that are the images captured at a first viewpoint and a second viewpoint, respectively, and first position information and second position information that are the position information of the first viewpoint and the second viewpoint, respectively; a measurement unit that measures a distance between the first position and the second position based on the estimation result; An information processing device comprising: (2) A detection unit for detecting a line of sight of a user is provided, The estimation unit is estimating the first position based on the line of sight detection result at the first viewpoint, the first image, and the first position information; estimating the second position based on the line of sight detection result at the second viewpoint, the second image, and the second position information; The information processing device according to (1). (3) The estimation unit is The first position is estimated based on the designation by the line of sight from the first viewpoint; The second position is estimated based on the designation by the line of sight from the second viewpoint. The information processing device according to (2). (4) a display control unit that displays a virtual object on a predetermined display unit in response to a detection result of the line of sight; The estimation unit is The first position is estimated based on a designation by the virtual object displayed according to a line of sight from the first viewpoint; The second position is estimated based on a designation by the virtual object displayed according to a line of sight from the second viewpoint. The information processing device according to (2). (5) The information processing device described in (1), wherein the estimation unit estimates at least one of the first position and the second position based on a specification corresponding to a detection result of contact or proximity to a real object in real space by a specified detection unit. (6) a display control unit that displays a virtual object on a predetermined display unit according to a recognition result of a real object in a real space; The estimation unit estimates at least one of the first position and the second position based on designation by the virtual object. The information processing device according to (1). (7) The information processing device according to (1), wherein the estimation unit estimates at least one of the first position and the second position based on a specification corresponding to a recognition result of an object in real space. (8) The information processing device according to (1), further comprising a display control unit that causes a predetermined display unit to display information based on the distance measurement result. (9) The information processing device according to (8), wherein the display control unit causes the display information indicating the distance measurement result to be displayed. (10) The information processing device according to (8) or (9), wherein the display control unit causes the display information indicating a scale in real space to be displayed based on a result of the distance measurement. (11) The information processing device according to any one of (8) to (10), wherein the display control unit causes the display information to be displayed in a size according to a result of the distance measurement. (12) The information processing device according to any one of (1) to (11), wherein the measurement unit estimates a distance of a path between the first position and the second position, passing through a third position specified between the first position and the second position. (13) The information processing device according to any one of (1) to (11), wherein the measurement unit measures, as the distance, a length of a path specified between the first position and the second position. (14) The information processing device according to any one of (1) to (13), wherein the acquisition unit acquires the first image and the first position information, and the second image and the second position information from external devices different from each other. (15) Acquiring an image captured by a predetermined imaging unit and position information based on at least one of a position and an orientation of the imaging unit; The processor: estimating a first position and a second position in a real space based on a first image and a second image which are the images captured at a first viewpoint and a second viewpoint, respectively, and first position information and second position information which are the position information of the first viewpoint and the second viewpoint, respectively; measuring a distance between the first location and the second location based on the estimation result; An information processing method comprising: (16) On the computer, Acquiring an image captured by a predetermined imaging unit and position information based on at least one of a position and an orientation of the imaging unit; estimating a first position and a second position in a real space based on a first image and a second image which are the images captured at a first viewpoint and a second viewpoint, respectively, and first position information and second position information which are the position information of the first viewpoint and the second viewpoint, respectively; measuring a distance between the first location and the second location based on the estimation result; A recording medium on which a program for executing the above is recorded. [Explanation of symbols]

[0253] 1, 2, 3 Information Processing System 10. Information processing device 101 Depth detection unit 103 Position and orientation estimation unit 105 Line of Sight Detection Unit 107 Point of interest detection unit 109 Trigger detection section 111 Reference point estimation part 113 Distance calculation unit 115 Output control section 20 Input / Output Devices 201 First Imaging Unit 203 Second Imaging Unit 205 Input section 207 Operation section 209 Sound collection section 30 Terminal Equipment 301 Imaging unit 303 Input section 305 Output section 40 Information processing device 401 Depth detection unit 403 Position and orientation estimation unit 405 Trigger detector 407 Reference point estimation section 409 Distance Calculation Unit 411 Output control section 50 Information processing device 501 Depth detection unit 503 Position and orientation estimation unit 505 Object recognition unit 507 Trigger detector 509 Reference point estimation part 511 Distance Calculation Unit 513 Output control section

Claims

1. A first information processing device; A second information processing device different from the first information processing device; An information processing system comprising: The second information processing device has a control unit, The control unit is determining either a first measurement mode or a second measurement mode based on a first user operation input from the first information processing device; When the first measurement mode is determined, control the display unit to set virtual reference points based on a second user operation and display a first virtual object indicating a distance between at least two of the virtual reference points; or When the second measurement mode is selected, controlling the display unit to display a second virtual object based on a third user operation. Information processing system.

2. The control unit: controlling the display unit to superimpose a third virtual object extending from the first information processing device on a real space; The information processing system according to claim 1 .

3. The control unit is controlling the display unit to display a fourth virtual object connecting at least two of the virtual reference points when the first measurement mode is determined; The information processing system according to claim 1 .

4. The fourth virtual object is a strip. The information processing system according to claim 3 .

5. The fourth virtual object is displayed together with a virtual scale. The information processing system according to claim 3 .

6. The control unit is controlling the display unit to display a fifth virtual object indicating an area of ​​a region surrounded by at least three of the virtual reference points when the first measurement mode is determined; The information processing system according to claim 1 .

7. the third virtual object is linear; The information processing system according to claim 2 .

8. The control unit is When the second measurement mode is determined, a position or a size of the second virtual object is changed based on the third user operation. The information processing system according to claim 1 .

9. The control unit is controlling the display unit to display a sixth virtual object indicating at least one of a length, a width, or a depth of the second virtual object when the second measurement mode is determined; The information processing system according to claim 1 .

10. The second user operation is a predetermined operation for setting the virtual reference point. The information processing system according to claim 1 .

11. The third user operation is a predetermined operation for superimposing the second virtual object on a real space. The information processing system according to claim 1 .

12. A first information processing device; A second information processing device different from the first information processing device; An information processing method in an information processing system comprising: determining either a first measurement mode or a second measurement mode based on a first user operation input from the first information processing device; When the first measurement mode is determined, control the display unit to set virtual reference points based on a second user operation and display a first virtual object indicating a distance between at least two of the virtual reference points; or When the second measurement mode is selected, controlling the display unit to display a second virtual object based on a third user operation. Information processing methods.

13. The second information processing device controls the display unit so as to superimpose a third virtual object extending from the first information processing device on a real space. The information processing method according to claim 12.

14. controlling the display unit to display a fourth virtual object connecting at least two of the virtual reference points when the first measurement mode is determined; The information processing method according to claim 12.

15. The fourth virtual object is a strip. The information processing method according to claim 14.

16. The fourth virtual object is displayed together with a virtual scale. The information processing method according to claim 14.

17. controlling the display unit to display a fifth virtual object indicating an area of ​​a region surrounded by at least three of the virtual reference points when the first measurement mode is determined; The information processing method according to claim 12.

18. the third virtual object is linear; The information processing method according to claim 13.

19. When the second measurement mode is determined, a position or a size of the second virtual object is changed based on the third user operation. The information processing method according to claim 12.

20. controlling the display unit to display a sixth virtual object indicating at least one of a length, a width, or a depth of the second virtual object when the second measurement mode is determined; The information processing method according to claim 12.

21. A first information processing device; A second information processing device different from the first information processing device; An information processing program for causing a computer to execute an information processing method in an information processing system comprising: The information processing method includes: determining either a first measurement mode or a second measurement mode based on a first user operation input from the first information processing device; When the first measurement mode is determined, control the display unit to set virtual reference points based on a second user operation and display a first virtual object indicating a distance between at least two of the virtual reference points; or When the second measurement mode is selected, controlling the display unit to display a second virtual object based on a third user operation. Information processing program for.

Images