Information processing system and information processing method

Abstract

[Problem] To highly precisely estimate a relative pose between devices without limiting movement of a user. [Solution] An information processing system is provided that comprises: an estimation unit that estimates a relative pose between a first device and a second device, on the basis of a first image captured by the first device attached to the head of a user, and a second image captured by the second device attached to a body part different from the head of the user; and a presentation control unit that controls presentation of information indicating a positional relation between the first device and the second device on the basis of the relative pose that is estimated.

Description

Information processing system and information processing method

[0001] The present disclosure relates to an information processing system and an information processing method.

[0002] In recent years, systems have been developed that use devices such as head-mounted displays (HMDs) to achieve Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), etc. Furthermore, for example, Patent Literature 1 discloses a technology that estimates in real time the transition of the position and orientation (hereinafter referred to as the pose) of a controller held by a user's hand based on an image captured by a camera provided in the head-mounted display, and performs control based on the estimated pose of the controller.

[0003] International Publication No. 2020 / 110659

[0004] However, with the method disclosed in Patent Document 1, it is difficult to estimate the pose of the controller if the controller is out of the field of view of the head-mounted display.

[0005] According to one aspect of the present disclosure, there is provided an information processing system comprising: an estimation unit that estimates a relative pose between a first device and a second device based on a first image taken by a first device worn on a user's head and a second image taken by a second device worn on a part of the user different from the head; and a presentation control unit that controls the presentation of information indicating the positional relationship between the first device and the second device based on the estimated relative pose.

[0006] According to another aspect of the present disclosure, there is provided an information processing method including: a processor estimating a relative pose between a first device and a second device based on a first image taken by a first device worn on a user's head and a second image taken by a second device worn on a part of the user different from the head; and controlling the presentation of information indicating a positional relationship between the first device and the second device based on the estimated relative pose.

[0007] FIG. 1 is a diagram for explaining the Outside-In method. FIG. 2 is a diagram for explaining the Inside-Out method. FIG. 3 is a diagram for explaining an error in a relative pose estimated by the Inside-Out method. FIG. 4 is a diagram for explaining an example of a situation in which a relative pose cannot be estimated in the Inside-Out method. FIG. 5 is a block diagram for explaining an example of a functional configuration of a head mounted display 10 according to an embodiment of the present disclosure. FIG. 6 is a block diagram for explaining an example of a functional configuration of a controller 20 according to the embodiment. FIG. 7 is a diagram for explaining an overview of processing performed by an information processing system according to the embodiment. FIG. 8 is a diagram for explaining an example of information presentation control for each case according to the embodiment. FIG. 9 is a diagram for explaining an example of guidance in CASE: A-1 according to the embodiment. FIG. 10 is a diagram for explaining an example of superimposed display of a first image I1 and a second image I2 according to the embodiment. FIG. 11 is a flowchart for explaining an example of a processing flow of the information processing system according to the embodiment. FIG. 12 is a flowchart for explaining an example of a processing flow of the information processing system according to the embodiment. FIG. 13 is a block diagram for explaining an example of a hardware configuration of an information processing device 90 according to the embodiment.

[0008] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] In addition, in this specification and drawings, when multiple identical components are to be described separately, letters or the like may be added to the end of the reference numerals. On the other hand, when it is not necessary to distinguish between multiple identical components, the letters or the like may be omitted and a description common to all of the multiple identical components may be given.

[0010] The description will be given in the following order: 1. Embodiment 1.1. Background 1.2. System Configuration Example 1.3. Processing Overview 1.4. Processing Details 2. Hardware Configuration Example 3. Summary

[0011] <1. Embodiments> <<1.1. Background>> In a system using AR, VR, or MR, for example, a virtual space is constructed by estimating in real time the transition of the pose of a device such as a head-mounted display worn on the user's head, and changing the image presented to the user based on the estimated head movement.

[0012] Furthermore, as described above, by estimating the transition of the pose of a controller or the like held in the user's hand in real time and reflecting that pose in the virtual space, it is possible to realize an intuitive interface for hand actions (e.g., grasping an object) and various operations within the virtual space.

[0013] One method for estimating the pose of a controller (or the user's hand holding the controller) is the so-called Outside-In method, which estimates the pose of the controller based on an image of the controller captured by a camera provided in a head-mounted display, as disclosed in Patent Document 1, for example.

[0014] FIG. 1 is a diagram for explaining the Outside-In method.

[0015] In the Outside-In method, for example, as shown in FIG. 1, a camera provided in a head-mounted display 10 worn on the head of a user 50 captures an image of a controller 20 held in the hand of the user 50, and the relative pose of the controller 20 with respect to the head-mounted display 10 is directly estimated based on the captured image.

[0016] However, in the Outside-In method, the controller 20 needs to be within the angle of view AV1 of the camera provided in the head-mounted display 10, so the range in which the hand holding the controller 20 can move is limited.

[0017] On the other hand, the pose of the controller 20 can also be estimated based on an image captured by a camera provided in the controller 20 (a so-called inside-out method).

[0018] FIG. 2 is a diagram for explaining the Inside-Out method.

[0019] In the Inside-Out method, for example, as shown in Figure 2, the pose of the head-mounted display 10 is estimated based on an image taken by a camera provided in the head-mounted display 10, and the pose of the controller 20 is estimated based on an image taken by a camera provided in the controller 20.

[0020] In this case, by estimating the pose of the head-mounted display 10 and the pose of the controller 20 based on the same environmental map 60 (see Figures 3, 4, etc.) in addition to the above images, it is possible to indirectly determine the relative pose of the controller 20 with respect to the head-mounted display 10.

[0021] As shown in the figure, with the Inside-Out method, it is not necessary to fit the controller 20 within the angle of view AV1 of the camera provided in the head-mounted display 10, and it is also not necessary to fit the head-mounted display 10 within the angle of view AV2 of the camera provided in the controller 20. Therefore, the Inside-Out method has the advantage that the range in which the hand holding the controller 20 can move is not limited.

[0022] However, in the Inside-Out method, the relative pose of the controller 20 with respect to the head mounted display 10 is estimated based on the pose of the head mounted display 10 and the pose of the controller 20. For this reason, the error in the relative pose estimated by the Inside-Out method may appear as a sum of the error in the pose of the head mounted display 10 and the error in the pose of the controller 20.

[0023] FIG. 3 is a diagram for explaining an error in the relative pose estimated by the Inside-Out method.

[0024] 3, it is assumed that the pose of the head mounted display 10 estimated based on an image taken by a camera provided in the head mounted display 10 and the environmental map 60 has an error ε1. It is also assumed that the pose of the controller 20 estimated based on an image taken by a camera provided in the controller 20 and the environmental map 60 has an error ε2.

[0025] In this case, the error in the relative pose estimated based on the pose of the head mounted display 10 and the pose of the controller 20 is ε1+ε2.

[0026] Furthermore, the Inside-Out method has a problem not found in the Outside-In method, in that if it is not possible to estimate either the pose of the head-mounted display 10 or the pose of the controller 20 relative to the common environmental map 60, it is also not possible to estimate the relative pose.

[0027] FIG. 4 is a diagram for explaining an example of a situation in which a relative pose cannot be estimated using the Inside-Out method.

[0028] Figure 4 shows an example of a case where the pose of the controller 20 can be estimated based on an image taken by a camera equipped in the controller 20 and the environmental map 60, but the pose of the head-mounted display 10 cannot be estimated based on an image taken by a camera equipped in the head-mounted display 10 and the environmental map 60 due to some factor.

[0029] In this case, the relative pose also cannot be estimated. Note that, contrary to the example shown in Fig. 4, if the pose of the head mounted display 10 can be estimated but the pose of the controller 20 cannot be estimated, the relative pose also cannot be estimated.

[0030] The technical concept of one embodiment of the present disclosure was conceived with the above points in mind, and makes it possible to accurately estimate the relative pose between devices without restricting the movement of the user.

[0031] An example of the configuration of an information processing system that achieves the above will be described in detail below.

[0032] <<1.2. System Configuration Example>> An information processing system according to an embodiment of the present disclosure includes at least a head-mounted display 10 and a controller 20.

[0033] FIG. 5 is a block diagram showing an example of the functional configuration of the head mounted display 10 according to this embodiment.

[0034] The head-mounted display 10 according to the present embodiment is an example of a first device.

[0035] As shown in FIG. 5, the head mounted display 10 according to this embodiment includes a control unit 110, a camera 120, a display unit 130, an operation receiving unit 140, a storage unit 150, a communication unit 160, and the like.

[0036] (Control Unit 110) The control unit 110 according to this embodiment controls the operation of each component included in the head mounted display 10.

[0037] The functions of the control unit 110 according to this embodiment are realized by cooperation between various processors such as a central processing unit (CPU) and a graphics processing unit (GPU) and a memory.

[0038] The control unit 110 according to this embodiment may operate as, for example, an estimation unit, a presentation control unit, a first pose estimation unit, and a generation unit.

[0039] (Camera 120) The camera 120 according to this embodiment captures an image of the environment around the head mounted display 10 under the control of the control unit 110.

[0040] (Display Unit 130) The display unit 130 according to this embodiment displays various types of visual information under the control of the control unit 110.

[0041] For example, the control unit 110 creates a virtual space by changing the image displayed on the display unit 130 based on the pose of the head-mounted display 10 (which can also be said to be the pose of the user's head) estimated in real time.

[0042] Furthermore, for example, the control unit 110 changes the image displayed on the display unit 130 based on the relative pose of the controller 20 with respect to the head-mounted display 10 (which can also be said to be the relative pose of the hand with respect to the user's head), thereby realizing an intuitive interface for actions such as grasping objects in the virtual space and various operations.

[0043] (Operation Acceptor 140) The operation acceptor 140 according to this embodiment accepts various operations by the user. The operation acceptor 140 may include, for example, a button, a switch, a lever, and the like.

[0044] (Storage Unit 150) The storage unit 150 according to this embodiment stores various types of information used by the head mounted display 10.

[0045] For example, the storage unit 150 according to this embodiment stores an environmental map 60 .

[0046] (Communication Unit 160) The communication unit 160 according to this embodiment communicates information with other devices including at least the controller 20. Any communication standard may be adopted for this information communication.

[0047] An example of the functional configuration of the head mounted display 10 according to this embodiment has been described above. Next, an example of the functional configuration of the controller 20 according to this embodiment will be described.

[0048] FIG. 6 is a block diagram showing an example of the functional configuration of the controller 20 according to this embodiment.

[0049] The controller 20 according to this embodiment is an example of a second device.

[0050] The controller 20 according to this embodiment is worn on a part of the head other than the head of the user wearing the head-mounted display 10 .

[0051] For example, the controller 20 according to this embodiment may be held in the user's hand.

[0052] Furthermore, for example, the controller 20 according to the present embodiment may be attached to the user's fingers, wrist, toes, ankle, etc. When the controller 20 is attached to the user's toes, ankle, etc., it becomes possible to detect actions such as operations performed with the feet and kicks.

[0053] As shown in FIG. 6, the controller 20 according to this embodiment includes a control unit 210, a camera 220, an operation receiving unit 240, a storage unit 250, a communication unit 260, and the like.

[0054] (Control Unit 210) The control unit 210 according to this embodiment controls the operation of each component included in the controller 20.

[0055] The functions of the control unit 210 according to this embodiment are realized by cooperation between various processors such as a CPU and a GPU and a memory.

[0056] The control unit 210 according to this embodiment may operate as, for example, a second pose estimation unit.

[0057] (Camera 220) The camera 220 according to this embodiment captures an image of the environment around the controller 20 under the control of the control unit 210.

[0058] (Operation Acceptor 240) The operation acceptor 240 according to this embodiment accepts various operations by the user. The operation acceptor 240 may include, for example, a button, a switch, a lever, or the like.

[0059] (Storage Unit 250) The storage unit 250 according to this embodiment stores various types of information used by the controller 20.

[0060] For example, the storage unit 250 according to this embodiment stores an environmental map 60 .

[0061] (Communication Unit 260) The communication unit 260 according to this embodiment performs information communication with other devices including at least the head mounted display 10. Any communication standard may be adopted for the information communication.

[0062] An example of the functional configuration of the controller 20 according to this embodiment has been described above.

[0063] Note that the functional configuration of the head-mounted display 10 described using Figure 5 and the functional configuration of the controller 20 described using Figure 6 are merely examples, and the functional configurations of the head-mounted display 10 and controller 20 according to this embodiment are not limited to these examples.

[0064] For example, the head-mounted display 10 and the controller 20 may further include an audio output unit, a vibration output unit, and the like.

[0065] The functional configuration of the head-mounted display 10 and the controller 20 can be flexibly modified depending on the specifications, operation, and the like.

[0066] <<1.3. Processing Overview>> Next, an overview of the processing performed by the information processing system according to this embodiment will be described with reference to FIG.

[0067] As shown in Figure 7, the information processing system of this embodiment generates an environmental map 60 based on, for example, an image captured by a camera 120 provided in the head-mounted display 10 (hereinafter also referred to as the first image) (S101).

[0068] The generation of the environmental map 60 in step S101 may be performed by, for example, the control unit 110 of the head-mounted display 10.

[0069] Next, the information processing system according to this embodiment estimates the pose of the head mounted display 10 (hereinafter also referred to as the first pose) based on the first image and the environmental map 60 (S102).

[0070] The first pose estimation in step S102 may be performed by, for example, the control unit 110 of the head-mounted display 10.

[0071] In addition, the information processing system according to this embodiment estimates the pose of the controller 20 (hereinafter also referred to as the second pose) based on an image (hereinafter also referred to as the second image) taken by the camera 220 provided in the controller 20 and the environmental map 60 (S103).

[0072] The second pose estimation in step S103 may be performed by, for example, the control unit 210 of the controller 20.

[0073] Next, the information processing system according to this embodiment estimates a relative pose between the head-mounted display 10 and the controller 20 based on the first pose estimated in step S102 and the second pose estimated in step S103 (S104).

[0074] Hereinafter, the first pose and the second pose will be collectively referred to as “both poses.” The relative pose estimation based on the two poses in step S104 may be performed by the control unit 110 of the head-mounted display 10, for example.

[0075] Furthermore, the information processing system according to this embodiment estimates a relative pose between the head mounted display 10 and the controller 20 based on the first image and the second image (S105).

[0076] Hereinafter, the first image and the second image will be collectively referred to as “both images.” The relative pose estimation based on both images in step S105 may be performed by the control unit 110 of the head-mounted display 10, for example.

[0077] Next, the information processing system of this embodiment determines a relative pose to be used to control the presentation of information indicating the positional relationship between the head-mounted display 10 and the controller 20 based on at least one of the relative pose estimated based on both poses in step S104 and the relative pose estimated based on both images in step S105 (S106).

[0078] In step S106, the information processing system according to this embodiment can detect, for example, an error in the relative pose estimated based on both poses, based on both images.

[0079] The premise is that if the error in the relative pose estimated based on both poses is sufficiently small, then when the first image is transformed using that relative pose, it should match the second image.

[0080] In order to achieve a perfect match, the distance to the subject must be known. If the distance is unknown or contains an error, a perfect match is not achieved, but rather a match is achieved by finding corresponding points on the epipolar line. In either case, however, if the error in the relative pose estimated based on both poses is sufficiently small, then the first image will be matched with the second image when transformed using that relative pose.

[0081] On the other hand, if the error in the relative pose estimated based on both poses is large, a discrepancy in consistency will appear, for example, the corresponding points will not be on the epipolar line, etc. The information processing system according to this embodiment can determine the magnitude of the error in the relative pose estimated based on both poses by checking for the presence or absence of such a discrepancy.

[0082] If the information processing system according to this embodiment determines that the error in the relative pose estimated based on both poses is large, it may perform processing such as, for example, performing presentation control to display the error in a way that makes it less noticeable, performing presentation control to guide the user's movements to reduce the error, or performing presentation control to prompt calibration.

[0083] Furthermore, the information processing system according to this embodiment may correct an error in the relative pose estimated based on both poses, based on the relative pose estimated based on both images. Furthermore, the information processing system according to this embodiment may use the corrected relative pose to present information indicating the positional relationship between the head-mounted display 10 and the controller 20.

[0084] Since the distance between the subjects in both images is unknown, the relative pose estimated in step S105 does not have a unique scale (actual physical length), but rather has an indefinite position vector (orientation) and scale, i.e., five of the six degrees of freedom for position and orientation are determined.

[0085] Therefore, the information processing system according to this embodiment can correct errors in the relative pose estimated based on both poses for the above five degrees of freedom based on the relative pose estimated based on both images.

[0086] The processing in step S106 may be performed by, for example, the control unit 110 of the head-mounted display 10.

[0087] Next, the information processing system according to this embodiment controls the presentation of information indicating the positional relationship between the head-mounted display 10 and the controller 20 based on the relative pose determined in step S106 (S107).

[0088] The control of presentation of the information indicating the positional relationship in step S107 may be performed by, for example, the control unit 110 of the head-mounted display 10.

[0089] The above has provided an overview of the processing performed by the information processing system.

[0090] The entity that performs each of the processes in steps S101 to S107 is not limited to the configuration exemplified above.

[0091] The subject that performs each of the processes in steps S101 to S107 may be the control unit 110 of the head-mounted display 10 or the control unit 210 of the controller 20.

[0092] Furthermore, if the information processing system includes a computer different from the head-mounted display 10 and the controller 20, the subject may be the computer.

[0093] Furthermore, although the above description has been given of the case where the environmental map 60 is generated based on a first image, the environmental map 60 may also be generated based on a second image.

[0094] <<1.4. Processing Details>> Next, the processing performed by the information processing system according to this embodiment will be described in more detail.

[0095] As explained in 1.3. Processing Overview above, the information processing system according to this embodiment is capable of estimating a relative pose based on both images. Therefore, even if one or both of the two poses cannot be estimated, the information processing system according to this embodiment can present information indicating the positional relationship between the head-mounted display 10 and the controller 20.

[0096] However, the information that can be presented may change depending on the estimated conditions of the first and second poses and the contents of the first and second images.

[0097] FIG. 8 is a diagram for explaining an example of information presentation control for each case according to this embodiment.

[0098] CASE: A-1 is a case where both poses cannot be estimated and one of the cameras does not capture the other device (the controller 20 is not captured in the first image and the head-mounted display 10 is not captured in the second image).

[0099] In this case, it is possible to estimate the relative pose based on the first and second images, but the relative pose has an indefinite scale.

[0100] However, even if the scale is indefinite, the orientation of the controller 20 relative to the head-mounted display 10 can be estimated, so in CASE: A-1, for example, control may be performed to present only the orientation of the controller 20.

[0101] CASE: A-2 is a case where both poses cannot be estimated and one of the cameras captures the other device (the controller 20 is captured in the first image, or the head-mounted display 10 is captured in the second image).

[0102] In CASE: A-2, similarly to CASE: A-1, it is possible to estimate the relative pose based on the first and second images, but the scale of the relative pose is indefinite.

[0103] However, in the case of CASE: A-2, it is possible to roughly estimate the scale based on the size of the controller 20 shown in the first image or the size of the head-mounted display 10 shown in the second image.

[0104] Therefore, in CASE: A-2, control may be performed to present the position of the controller 20 using an approximate scale.

[0105] As described above, when one of the cameras captures the other device, the position of the controller 20 can be presented by roughly estimating the scale. Therefore, in CASE: A-1, information may be presented to guide the other device to be captured by one of the cameras.

[0106] FIG. 9 is a diagram showing an example of guidance in CASE: A-1.

[0107] If vector a in the line of sight direction of camera 120 is brought closer to position vector b from head-mounted display 10 toward controller 20, controller 20 will be within angle of view AV1 of camera 120 (i.e., controller 20 will appear in the first image). Therefore, for example, control unit 110 may cause display unit 130 to display an arrow corresponding to vector b-a, as illustrated in FIG.

[0108] Furthermore, as illustrated in FIG. 9 , control unit 110 may cause display unit 130 to display a message guiding the user to face the direction of controller 20 .

[0109] In addition, the control unit 110 may cause the display unit 130 to display a virtual object using computer graphics (CG) that matches the world view of the application (e.g., a game, navigation, etc.), and may provide natural guidance to the user, such as having the user follow the virtual object.

[0110] The guidance display as described above can increase the possibility that the controller 20 will appear in the first image.

[0111] However, even in CASE: A-1, it is possible to present the position of the controller 20 by using the general (average) distance between the head and the hand.

[0112] The description will continue with reference to FIG.

[0113] CASE: B is the case where only one of the two poses can be estimated.

[0114] In this case, it is not possible to estimate a relative pose based on both poses, but the scale of one of the poses that can be estimated can also be obtained from the environment map 60. Therefore, in CASE: B, it is possible to estimate a relative pose including the scale based on the one pose that can be estimated, the first image, and the second image.

[0115] In CASE: B, the position of the controller 20 relative to the head-mounted display 10 may be presented based on the relative pose including the scale estimated as described above.

[0116] Also, CASE: C is the case where both poses can be estimated.

[0117] In this case, it is possible to estimate a relative pose including scale based on both poses, and to perform correction using the relative pose estimated based on both images.

[0118] In CASE: C, the position of the controller 20 relative to the head-mounted display 10 may be presented based on the relative pose including the scale estimated and corrected as described above.

[0119] The above describes examples of information presentation control for each case according to this embodiment.

[0120] In order to estimate the relative pose based on both images, both images must have a common field of view (the same area must be captured in both images).

[0121] Therefore, the information processing system according to this embodiment (for example, the control unit 110 provided in the information processing system) may control the presentation of information that guides the two images to have a common field of view when the two images do not have a common field of view.

[0122] When both poses can be estimated, it is possible to calculate the line-of-sight vector of camera 120 based on the first pose, and it is possible to calculate the line-of-sight vector of camera 220 based on the second pose. Therefore, it is possible to present information that guides the two images to have a common field of view using a method equivalent to the method described using FIG.

[0123] Furthermore, whether or not the two images have a common field of view is obvious from looking at both images. Therefore, the information processing system according to this embodiment (for example, the control unit 110 included in the information processing system) may display the first image I1 and the second image I2 in a superimposed manner on the display unit 130, as shown in FIG.

[0124] FIG. 10 is a diagram showing an example of a superimposed display of a first image I1 and a second image I2 according to this embodiment.

[0125] Next, the processing flow of the information processing system according to this embodiment will be described in more detail.

[0126] 11A and 11B are flowcharts showing an example of the flow of processing in the information processing system according to this embodiment.

[0127] It should be noted that FIGS. 11A and 11B are connected to each other by the circled letters.

[0128] In the example shown in Figures 11A and 11B, first, the control unit 110 estimates the pose (first pose) of the head mounted display 10 based on an image (first image) captured by the camera 120 (S201).

[0129] Furthermore, the control unit 210 estimates the pose of the controller 20 (second pose) based on the image (second image) captured by the camera 220 (S202).

[0130] Next, the control unit 110 determines whether or not both poses have been estimated (S203).

[0131] When it is determined that both poses have been estimated (S203: YES), the control unit 110 estimates a relative pose based on both poses (S204).

[0132] Furthermore, when it is determined that both poses have been estimated (S203: YES), the control unit 110 determines whether or not both images have a common field of view (S205).

[0133] If the control unit 110 determines that both images have a common field of view (S205: YES), it estimates a relative pose based on both images (S206).

[0134] Next, the control unit 110 corrects the relative pose estimated based on both poses in step S204 with the relative pose estimated based on both images in step S206, and determines the corrected relative pose as the relative pose to be used for presenting information indicating the positional relationship between the head-mounted display 10 and the controller 20 (S207).

[0135] In addition, if the control unit 110 determines that the two images do not have a common field of view (S205: NO), it may determine the relative pose estimated based on both poses in step S204 as the relative pose to be used for presenting information indicating the above-mentioned positional relationship, and skip step S207.

[0136] The control unit 110 controls the presentation of the position of the controller 20 based on the determined relative pose (S208).

[0137] Furthermore, the control unit 110 may detect various operations in the virtual space based on the determined relative pose.

[0138] On the other hand, when it is determined that both poses have not been estimated (S203: NO), the control unit 110 determines whether one of the two poses has been estimated (S209).

[0139] When the control unit 110 determines that one of the poses has been estimated (S209: YES), it determines whether or not the two images have a common field of view (S210).

[0140] If the control unit 110 determines that both images have a common field of view (S210: YES), it estimates a relative pose based on one of the estimated poses and both images (S211), and determines the estimated relative pose as the relative pose to be used for presenting information indicating the positional relationship.

[0141] After step S211, the control unit 110 controls the presentation of the position of the controller 20 based on the determined relative pose (S208).

[0142] On the other hand, if the control unit 110 determines that neither of the poses has been estimated (S209: NO), it determines whether the two images have a common field of view (S212).

[0143] If the control unit 110 determines that both images have a common field of view (S212: YES), it estimates a relative pose based on both images (S213) and determines the estimated relative pose as the relative pose to be used for presenting information indicating the positional relationship.

[0144] Next, the control unit 110 determines whether the scale can be roughly calculated (S214).

[0145] If the controller 20 is shown in the first image or the head-mounted display 10 is shown in the second image, the control unit 110 determines that the scale can be estimated (S214: YES), and estimates the scale of the relative pose estimated in step S213 based on the size of the device shown in the image (S215).

[0146] After step S215, the control unit 110 controls the presentation of the position of the controller 20 based on the determined relative pose (S208).

[0147] If the control unit 110 determines that the scale cannot be roughly calculated (S214: NO), it controls the presentation of the orientation of the controller 20 based on the determined relative pose (S216).

[0148] That is, when the determined relative pose does not have a scale, the control unit 110 may control the presentation of information indicating the orientation of the controller 20 relative to the head-mounted display 10 .

[0149] In contrast, when the determined relative pose has a scale, the control unit 110 controls the presentation of information indicating the position of the controller 20 with respect to the head-mounted display 10 as described above.

[0150] Next, the control unit 110 performs presentation control to guide the other device to appear in the image (so that the controller 20 appears in the first image, or so that the head-mounted display 10 appears in the second image) (S217).

[0151] On the other hand, if the control unit 110 determines that the two images do not have a common field of view (S205: NO, S210: NO, or S212: NO), it performs presentation control to guide the two images to have a common field of view (S218).

[0152] An example of the processing flow of the information processing system according to this embodiment has been described above.

[0153] According to the above series of processes, it is possible to estimate the relative pose between devices with high accuracy without restricting the user's movements.

[0154] In the above, we have described a case where information indicating the positional relationship between the head-mounted display 10 and the controller 20 is presented mainly using visual information, but the information indicating the positional relationship may also be presented using sound, vibration, etc.

[0155] Furthermore, although the above mainly describes a case where the head-mounted display 10 and the controller 20 correspond to each other in a one-to-one relationship, the head-mounted display 10 and the controller 20 may also correspond to each other in a one-to-N relationship.

[0156] Furthermore, the combination of devices according to this embodiment is not limited to the combination of the head-mounted display 10 and the controller 20.

[0157] For example, it is conceivable that head-mounted displays 10 worn by multiple users each perform pose estimation based on the same environmental map 60 and share a virtual space (for example, an application in which a virtual object placed in a certain location from the perspective of one user appears to be in the same location from the perspective of another user).

[0158] In the above case, similar to the error in the relative pose of the controller 20 with respect to the head mounted display 10, there may be a problem of an error in the relative pose between the head mounted displays 10, and a problem that the relative pose cannot be determined if one of the poses cannot be estimated. However, by applying the technical idea according to one embodiment of the present disclosure, a similar effect can be expected for the estimation of the relative pose between the head mounted displays 10.

[0159] 2. Hardware Configuration Example Next, a hardware configuration example common to the head-mounted display 10 and the controller 20 according to an embodiment of the present disclosure will be described.

[0160] 12 is a block diagram showing an example of a hardware configuration of an information processing device 90 according to an embodiment of the present disclosure. The information processing device 90 may be a device having a hardware configuration equivalent to that of each of the above-described devices.

[0161] 12 , the information processing device 90 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the information processing device 90 may include further components other than those shown here.

[0162] (Processor 871) The processor 871 functions, for example, as an arithmetic processing device or control device, and controls the overall operation of each component or part of it based on various programs recorded in the ROM 872, RAM 873, storage 880, or removable storage medium 901.

[0163] (ROM 872, RAM 873) The ROM 872 is a means for storing programs to be read into the processor 871, data to be used for calculations, etc. The RAM 873 temporarily or permanently stores, for example, the programs to be read into the processor 871 and various parameters that change as appropriate when the programs are executed.

[0164] (Host bus 874, bridge 875, external bus 876, interface 877) The processor 871, ROM 872, and RAM 873 are connected to one another via, for example, a host bus 874 that is capable of high-speed data transmission. On the other hand, the host bus 874 is connected to, for example, an external bus 876 that has a relatively low data transmission speed via a bridge 875. Furthermore, the external bus 876 is connected to various components via an interface 877.

[0165] (Input Device 878) For example, a mouse, keyboard, touch panel, button, switch, lever, etc. are used as the input device 878. Furthermore, a remote controller (hereinafter referred to as a remote control) capable of transmitting control signals using infrared rays or other radio waves may also be used as the input device 878. The input device 878 also includes an audio input device such as a microphone.

[0166] (Output Device 879) The output device 879 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), or an organic EL display, an audio output device such as a speaker or a headphone, a printer, a mobile phone, a facsimile, etc. The output device 879 according to the present disclosure also includes various vibration devices capable of outputting tactile stimulation.

[0167] (Storage 880) The storage 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage 880.

[0168] (Drive 881) The drive 881 is a device that reads information recorded on a removable storage medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable storage medium 901.

[0169] (Removable storage medium 901) The removable storage medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable storage medium 901 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.

[0170] (Connection Port 882) The connection port 882 is a port for connecting an external device 902, such as a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal.

[0171] (Externally Connected Device 902) The externally connected device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0172] (Communication device 883) The communication device 883 is a communication device for connecting to a network, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), Wi-Fi (registered trademark) or Wireless USB (WUSB), a router for optical communication, a router for Asymmetric Digital Subscriber Line (ADSL), or a modem for various types of communication.

[0173] 3. Summary As described above, an information processing system according to an embodiment of the present disclosure includes an estimation unit that estimates a relative pose between a first device and a second device based on a first image captured by a first device worn on a user's head and a second image captured by a second device worn on a part of the user different from the head, and a presentation control unit that controls presentation of information indicating a positional relationship between the first device and the second device based on the estimated relative pose.

[0174] According to the above configuration, it is possible to estimate the relative pose between devices with high accuracy without restricting the movement of the user.

[0175] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0176] Furthermore, the steps of the processes described in this disclosure do not necessarily have to be processed in chronological order according to the order shown in the flowcharts or sequence diagrams. For example, the steps of the processes of each device may be processed in an order different from the order shown, or may be processed in parallel.

[0177] Furthermore, the series of processes performed by each device described in this disclosure may be realized by a program stored in a non-transitory computer-readable storage medium. Each program is, for example, loaded into RAM when executed by a computer and executed by a processor such as a CPU. The storage medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. The program may also be distributed, for example, via a network, without using a storage medium.

[0178] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0179] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing system comprising: an estimation unit that estimates a relative pose between a first device and a second device based on a first image captured by a first device worn on a user's head and a second image captured by a second device worn on a part of the user different from the head; and a presentation control unit that controls presentation of information indicating a positional relationship between the first device and the second device based on the estimated relative pose. (2) The information processing system according to (1), further comprising: a first pose estimation unit that estimates a pose of the first device based on the first image and an environmental map; and a second pose estimation unit that estimates a pose of the second device based on the second image and the environmental map, wherein the estimation unit further estimates the relative pose based on the pose of the first device and the pose of the second device. (3) The information processing system according to (2), wherein the estimation unit determines the relative pose to be used for presenting the information indicating the positional relationship based on at least one of the relative pose estimated based on the first image and the second image, or the relative pose estimated based on the pose of the first device and the pose of the second device. (4) The information processing system according to (3), wherein, when the pose of the first device and the pose of the second device have been estimated and the first image and the second image have a common field of view, the estimation unit determines the relative pose to be used for presenting the information indicating the positional relationship by correcting the relative pose estimated based on the pose of the first device and the pose of the second device with the relative pose estimated based on the first image and the second image.(5) The information processing system according to (3), wherein, when either the pose of the first device or the pose of the second device has been estimated and the first image and the second image have a common field of view, the estimation unit determines the relative pose estimated based on either the estimated pose of the first device or the pose of the second device, the first image, and the second image as the relative pose to be used for presenting information indicating the positional relationship between the first device and the second device. (6) The information processing system according to (3), wherein, when both the pose of the first device and the pose of the second device have not been estimated and the first image and the second image have a common field of view, the estimation unit determines the relative pose estimated based on the first image and the second image as the relative pose to be used for presenting information indicating the positional relationship between the first device and the second device. (7) The information processing system according to (6), wherein the estimation unit roughly estimates a scale related to the relative pose when the second device is shown in the first image or when the first device is shown in the second image. (8) The information processing system according to any one of (3) to (7), wherein the presentation control unit controls the presentation of information indicating the position of the second device relative to the first device when the relative pose determined by the estimation unit has a scale. (9) The information processing system according to any one of (3) to (7), wherein the presentation control unit controls the presentation of information indicating the orientation of the second device relative to the first device when the relative pose determined by the estimation unit does not have a scale. (10) The information processing system according to any one of (4) to (6), wherein the presentation control unit controls the presentation of information guiding the first image and the second image to have a common field of view when the first image and the second image do not have a common field of view. (11) The information processing system according to (10), wherein the presentation control unit displays the first image and the second image in a superimposed manner.(12) The information processing system according to (7), wherein the presentation control unit controls the presentation of at least one of information guiding the second device to appear in the first image or information guiding the first device to appear in the second image when both the pose of the first device and the pose of the second device have not been estimated, the first image and the second image have a common field of view, the second device is not shown in the first image, and the first device is not shown in the second image. (13) The information processing system according to any one of (2) to (12), further comprising: a generation unit that generates the environmental map based on the first image or the second image. (14) The information processing system according to any one of (1) to (13), wherein the first device includes a head-mounted display. (15) The information processing system according to (14), wherein the presentation control unit causes a display unit included in the first device to display information indicating a positional relationship between the first device and the second device. (16) The information processing system according to any one of (1) to (15), wherein the second device includes a controller worn on the user's hand or foot. (17) An information processing method, including: a processor estimating a relative pose between the first device and the second device based on a first image taken by a first device worn on the user's head and a second image taken by a second device worn on a part of the user different from the head, and controlling presentation of information indicating a positional relationship between the first device and the second device based on the estimated relative pose. (18) The information processing method according to (17), further including: estimating a pose of the first device based on the first image and an environmental map, estimating a pose of the second device based on the second image and the environmental map, and estimating the relative pose based on the pose of the first device and the pose of the second device.(19) The information processing method according to (18), further comprising: determining the relative pose used to present the information indicating the positional relationship based on at least one of the relative pose estimated based on the first image and the second image, or the relative pose estimated based on the pose of the first device and the pose of the second device.

[0180] REFERENCE SIGNS LIST 10 Head-mounted display 110 Control unit 120 Camera 130 Display unit 140 Operation reception unit 150 Storage unit 160 Communication unit 20 Controller 210 Control unit 220 Camera 240 Operation reception unit 250 Storage unit 260 Communication unit

Claims

1. An information processing system comprising: an estimation unit that estimates a relative pose between a first device and a second device based on a first image taken by a first device worn on a user's head and a second image taken by a second device worn on a part of the user different from the head; and a presentation control unit that controls the presentation of information indicating the positional relationship between the first device and the second device based on the estimated relative pose.

2. The information processing system of claim 1, further comprising: a first pose estimation unit that estimates the pose of the first device based on the first image and the environmental map; and a second pose estimation unit that estimates the pose of the second device based on the second image and the environmental map, wherein the estimation unit further estimates the relative pose based on the pose of the first device and the pose of the second device.

3. The information processing system of claim 2, wherein the estimation unit determines the relative pose to be used to present information indicating the positional relationship based on at least one of the relative pose estimated based on the first image and the second image, or the relative pose estimated based on the pose of the first device and the pose of the second device.

4. The information processing system of claim 3, wherein, when the pose of the first device and the pose of the second device have been estimated and the first image and the second image have a common field of view, the estimation unit determines the relative pose to be used in presenting information indicating the positional relationship by correcting the relative pose estimated based on the pose of the first device and the pose of the second device with the relative pose estimated based on the first image and the second image.

5. The information processing system of claim 3, wherein, when either the pose of the first device or the pose of the second device has been estimated and the first image and the second image have a common field of view, the estimation unit determines the relative pose estimated based on either the estimated pose of the first device or the pose of the second device, the first image, and the second image as the relative pose to be used for presenting information indicating the positional relationship between the first device and the second device.

6. The information processing system of claim 3, wherein, when both the pose of the first device and the pose of the second device have not been estimated and the first image and the second image have a common field of view, the estimation unit determines the relative pose estimated based on the first image and the second image as the relative pose to be used for presenting information indicating the positional relationship between the first device and the second device.

7. The information processing system according to claim 6, wherein the estimation unit roughly estimates a scale related to the relative pose when the second device is shown in the first image or when the first device is shown in the second image.

8. The information processing system according to claim 3, wherein the presentation control unit controls the presentation of information indicating the position of the second device relative to the first device when the relative pose determined by the estimation unit has a scale.

9. The information processing system according to claim 3, wherein the presentation control unit controls the presentation of information indicating the orientation of the second device relative to the first device when the relative pose determined by the estimation unit does not have a scale.

10. The information processing system of claim 4, wherein the presentation control unit controls the presentation of information that guides the first image and the second image to have a common field of view when the first image and the second image do not have a common field of view.

11. The information processing system according to claim 10, wherein the presentation control unit displays the first image and the second image in a superimposed manner.

12. The information processing system of claim 7, wherein the presentation control unit controls the presentation of at least one of information guiding the second device to appear in the first image or information guiding the first device to appear in the second image when both the pose of the first device and the pose of the second device have not been estimated, the first image and the second image have a common field of view, the second device is not shown in the first image, and the first device is not shown in the second image.

13. The information processing system according to claim 2, further comprising: a generation unit that generates the environmental map based on the first image or the second image.

14. The information processing system according to claim 1, wherein the first device includes a head-mounted display.

15. The information processing system according to claim 14, wherein the presentation control unit causes a display unit provided in the first device to display information indicating the positional relationship between the first device and the second device.

16. The information processing system according to claim 1, wherein the second device includes a controller attached to the user's hand or foot.

17. An information processing method comprising: a processor estimating a relative pose between a first device and a second device based on a first image taken by a first device worn on a user's head and a second image taken by a second device worn on a part of the user different from the head; and controlling the presentation of information indicating the positional relationship between the first device and the second device based on the estimated relative pose.

18. The information processing method of claim 17, further comprising: estimating a pose of the first device based on the first image and an environment map; estimating a pose of the second device based on the second image and the environment map; and estimating the relative pose based on the pose of the first device and the pose of the second device.

19. The information processing method of claim 18, further comprising: determining the relative pose used to present information indicating the positional relationship based on at least one of the relative pose estimated based on the first image and the second image, or the relative pose estimated based on the pose of the first device and the pose of the second device.

Images