Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

To enable an experiencing person to grasp the detection state of his / her hand when the hand of the experiencing person is detected in an area which is not within the visual field range of the experiencing person.SOLUTION: An information processor 200 is configured to communicate with a HMD100 including a display part 130 which is arranged so as to cover the eyes of an experiencing person and displays an experience image based on an actual space to the experiencing person. The information processing apparatus includes a detection unit 210 configured to detect a hand of an experiencing person based on information obtained by capturing an image of a range wider than a visual field range of the experiencing person in a real space, and a display control unit 231 configured to perform control to display, on a display unit 130, information indicating a predetermined state in which the hand of the experiencing person is detected by the detection unit 210 and the detected hand of the experiencing person is not within the visual field range of the experiencing person.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, in the fields of design and manufacturing, there has been a demand for shortening the time and cost required for evaluation using prototypes. For example, mixed reality (MR) systems have been introduced to evaluate ease of assembly and maintainability using design (shape and design) data created with CAD (computer-aided design) systems.

[0003] For example, head-mounted image display devices (HMDs) equipped with MR systems are classified into video see-through and optical see-through types. Here, a video see-through HMD generates a display image by superimposing an image of a virtual reality space generated according to the position and orientation of the camera on a captured image of a real space captured by a camera. Examples of the image of the virtual reality space include images of virtual reality objects and text information rendered using computer graphics (CG). The video see-through HMD then displays the generated display image as an experiential image on a display unit, allowing the user to experience a mixed reality space by visually recognizing the experiential image displayed on the display unit with the user's eyes. On the other hand, an optical see-through HMD displays an image of the virtual reality space generated according to the position and orientation of the user's head, while simultaneously displaying the real space in front of the HMD using a half mirror or the like so that the user can observe the mixed reality space.

[0004] In addition, MR systems can detect the user's hands from captured images and use the detection results to operate the CG. In this way, MR systems can use the user's hands as input devices. In addition, in order to detect the user's hands over a wide range in real space, the HMD may be equipped with a camera that captures hand detection images with a wider angle of view than the captured images presented to the user, in addition to the camera that captures the captured images.

[0005] Patent Document 1 relates to an HMD and describes a technology that uses an image from an imaging unit that captures real space, and notifies the user if a gesture made by the user's hands cannot be recognized from the image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-83916 Summary of the Invention [Problem to be solved by the invention]

[0007] As described above, when detecting the user's hands using a camera that captures a wider range than the captured image presented to the user, the user's hands may not appear in the captured image presented to the user, but may be detected. In this regard, Patent Document 1 does not take into consideration the case where the user's hands are not appear in the image presented to the user and the user's hands are detected in an area where the user is not visible. Therefore, Patent Document 1 has a problem in that when the user's hands are detected in an area that is not within the user's field of view, it is difficult for the user to understand the detection status of their own hands.

[0008] The present invention has been made in consideration of such problems, and aims to enable the user to understand the detection status of their own hand when their hand is detected in an area that is not within the user's field of vision. [Means for solving the problem]

[0009] The information processing device of the present invention is an information processing device configured to be able to communicate with a display device that is positioned to cover the eyes of the experiencer and has a display unit that displays an experience image based on real space to the experiencer, and has: a detection means that detects the experiencer's hand based on information obtained by imaging or measuring a range in the real space that is wider than the experiencer's field of view; and a display control means that, when the experiencer's hand is detected by the detection means and the detected experiencer's hand is not within the experiencer's field of view, controls the display unit to display information indicating that the specified state is occurring. [Effects of the Invention]

[0010] According to the present invention, when the user's hand is detected in an area that is not within the user's visual field, the user can understand the detection status of his or her own hand. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of an information processing system according to a first embodiment. [Figure 2] 2A to 2C are diagrams for explaining an example of a hand detection image captured by a hand detection image capturing section and a presentation image captured by a presentation image capturing section shown in FIG. 1. [Figure 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device according to a first embodiment. [Figure 4] 5 is a flowchart showing an example of a processing procedure in a control method for an information processing device according to the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of processing in step S104 of FIG. 4. [Figure 6] FIG. 10 is a diagram illustrating an example of a schematic configuration of an information processing system according to a second embodiment. [Figure 7] 10 is a flowchart showing an example of a processing procedure in a control method for an information processing device according to a second embodiment. [Figure 8] FIG. 8 is a diagram for explaining an example of processing in step S202 of FIG. 7. [Figure 9] FIG. 10 is a diagram illustrating an example of a schematic configuration of an information processing system according to a third embodiment and a first modification thereof. [Figure 10] 10 is a flowchart showing an example of a processing procedure in a control method for an information processing device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] (First embodiment) First, the first embodiment will be described.

[0014] Fig. 1 is a diagram showing an example of a schematic configuration of an information processing system 10-1 according to the first embodiment. As shown in Fig. 1, the information processing system 10-1 includes an HMD 100 and an information processing device 200. In this embodiment, the HMD 100 and the information processing device 200 are communicatively connected to each other so that data can be communicated between them via at least one of wired communication and wireless communication.

[0015] The HMD 100 is a display device that is disposed so as to cover the user's eyes and includes a display unit 130 that displays experiential images based on a real space to the user. Specifically, the HMD 100 is a head-mounted image display device that is worn on the user's head and allows the user to experience experiential images of a mixed reality space. Note that, although the present embodiment uses the HMD 100 as an example of a head-mounted image display device, other types of head-mounted image display devices may also be applied. Furthermore, the present embodiment is not limited to a head-mounted image display device, and other types of image display devices, such as a handheld image display device, may also be applied as long as they are an image display device that is disposed so as to cover the user's eyes and is visible to allow the user to experience a mixed reality space.

[0016] 1, the HMD 100 includes a hand detection image capturing unit 110, a presentation image capturing unit 120, and a display unit 130. Note that, although the HMD 100 in the first embodiment is described as an example assuming a video see-through type HMD, the present invention is not limited to this video see-through type HMD.

[0017] The display unit 130 displays an image of the mixed reality space as an experiential image based on the real space transmitted from the information processing device 200. In this embodiment, the display unit 130 may be configured to include two displays arranged corresponding to the left and right eyes of the user. In this case, an image of the mixed reality space for the left eye is displayed on the display corresponding to the user's left eye, and an image of the mixed reality space for the right eye is displayed on the display corresponding to the user's right eye.

[0018] The presentation image capturing unit 120 is an imaging device that captures images of real space, and includes a right capturing unit that captures images to be presented to the right eye of the user, and a left capturing unit that captures images to be presented to the left eye of the user. The right and left capturing units of the presentation image capturing unit 120 are attached in parallel, and sequentially transmit images of each frame constituting a moving image (images of real space) to the information processing device 200. Here, in this embodiment, the relative positional relationship between the right and left capturing units of the presentation image capturing unit 120 is stored. Furthermore, in this embodiment, the internal parameters of the right and left capturing units of the presentation image capturing unit 120 (such as the focal length, principal point, and angle of view of the capturing units) are also stored.

[0019] In order to detect the user's hands, which can function as input devices for the experience image, the hand detection image capture unit 110 is an image capture device that captures an image of a wider range than the real space captured by the presentation image capture unit 120. Specifically, in this embodiment, the hand detection image capture unit 110 captures an image of a wider range than the real space related to the experience image displayed on the display unit 130, that is, an image of a wider range than the field of view of the user wearing the HMD 100.

[0020] FIG. 2 is a diagram illustrating an example of a hand detection image 250 captured by the hand detection image capturing unit 110 and a presentation image 260 captured by the presentation image capturing unit 120 shown in FIG. 1. In the example shown in FIG. 2, the hand detection image 250 captured by the hand detection image capturing unit 110 has, for example, a horizontal angle of view of 60 degrees and a vertical angle of view of 100 degrees. In the example shown in FIG. 2, the presentation image 260 captured by the presentation image capturing unit 120 has, for example, a horizontal angle of view of 50 degrees and a vertical angle of view of 40 degrees. Here, the presentation image 260 captured by the presentation image capturing unit 120 is, for example, an image that serves as the basis for an experience image (an image in mixed reality space) displayed on the display unit 130. In addition, in this embodiment, the arrangement of the hand detection image capturing unit 110 and the presentation image capturing unit 120 allows them to be captured as at least a portion of the same area. In this embodiment, as shown in FIG. 2, it is preferable that the hand detection image 250 captured by the hand detection image capturing unit 110 is an image that includes the range of the presentation image 260 captured by the presentation image capturing unit 120 in real space and has a wider range than the range of the presentation image 260.

[0021] Here, we return to the explanation of Figure 1 again. The information processing device 200 is configured to be able to communicate with the HMD 100, which is an image display device. The information processing device 200 can be configured, for example, as a computer device such as a PC (personal computer), or a portable terminal device such as a smartphone or tablet terminal device. As shown in FIG. 1 , the information processing device 200 has a functional configuration of a detection unit 210, a determination unit 220, and a display control means 231 including a notification unit 230.

[0022] The detection unit 210 is a detection means for detecting the hands of the user wearing the HMD 100 based on a hand detection image, which is information obtained by capturing an image of a range wider than the field of view of the user in real space using the hand detection image capturing unit 110.

[0023] The determination unit 220 is a determination means that determines whether or not the experiencer's hand has been detected by the detection unit 210, and, when the experiencer's hand has been detected by the detection unit 210, determines whether or not the detected experiencer's hand is within the field of view of the experiencer. Here, in the example of Fig. 1, the determination unit 220 determines whether or not the experiencer's hand detected by the detection unit 210 is within the field of view of the experiencer based on whether or not the experiencer's hand detected by the detection unit 210 appears in the presentation image captured by the presentation image capturing unit 120.

[0024] The notification unit 230 is a notification means that, when a predetermined state occurs in which the user's hand is detected by the detection unit 210 and the detected user's hand is not within the user's field of view (not shown in the presented image), notifies the HMD 100 of information indicating that the predetermined state exists. Furthermore, the display control means 231, which includes the notification unit 230, controls the display unit 130 to display information indicating that the predetermined state exists in which the user's hand is detected by the detection unit 210 and the detected user's hand is not within the user's field of view.

[0025] 3 is a diagram showing an example of the hardware configuration of the information processing device 200 according to the first embodiment. As shown in FIG. 3, the information processing device 200 has a hardware configuration including a processor 310, a memory 320, a storage medium 330, an input I / F 340, and an output I / F 350.

[0026] 3, the processor 310 is, for example, a CPU, and controls the overall operation of the information processing device 200 and performs various processes. The memory 320 is, for example, a RAM, and temporarily stores programs, various types of information, etc. The storage medium 330 is a computer-readable storage medium, such as a hard disk or a CD-ROM, and stores programs, various types of information, etc., for the long term. In this embodiment, the program stored in the storage medium 330 is read into the memory 320. Then, the processor 310 executes the program read into the memory 320, thereby realizing the functional configuration of the display control means 231, which includes the detection unit 210, the determination unit 220, and the notification unit 230 shown in FIG. 1.

[0027] 3, the input I / F 340 inputs various types of information (including various types of images) input from the HMD 100, which is an external device, as input information (including various types of images) in a format processable by the information processing device 200. The output I / F 350 outputs various types of information (including various types of images) processed by the processor 310 as output information (including various types of images) to the HMD 100, which is an external device, in a format processable by the HMD 100.

[0028] 4 is a flowchart showing an example of a processing procedure in a control method for the information processing device 200 according to the first embodiment. In the processing of the flowchart shown in FIG. 4, when a user's hand is detected in an area that is not visible to the user wearing the HMD 100 (an area that is not within the user's visual field), the user is notified via the display unit 130 of the HMD 100 that a hand has been detected. This allows the user to understand the detection status of their own hand, which can function as an input device for the experiential image.

[0029] When the processing of the flowchart of FIG. 4 starts, first, in step S101 of FIG. 4, the detection unit 210 performs processing to detect the user's hands from the hand detection image captured by the hand detection image capturing unit 110. Here, in this embodiment, the method for detecting hands is not limited in any way. For example, if the hand joint points on the image are recognized by machine learning and the hand detection image capturing unit 110 is configured with a stereo camera, the hand may be detected by determining the depth by triangulation to determine the three-dimensional positions of the hand joint points in real space. In this case, the hand joint points may be, for example, the wrist, palm, or joint points of each finger. In addition, the three-dimensional positions of the hand joint points may be positions in a coordinate system based on the hand detection image capturing unit 110.

[0030] In this embodiment, the process detects the user's hands from hand detection images captured by hand detection image capturing unit 110, but the present invention is not limited to this form, and the process may be performed using other methods. For example, a hand detection image capturing unit 110 that does not have a stereo camera configuration but can also measure depth may be prepared, and the process may be performed to detect the user's hands from hand detection images captured by this hand detection image capturing unit 110.

[0031] Furthermore, the process of detecting the user's hand may be performed using a sensor capable of measuring three-dimensional position, such as an optical sensor or a magnetic sensor. When using such a sensor capable of measuring three-dimensional position, for example, markers may be attached to each joint point of the hand, and the sensor may measure the three-dimensional position of each joint point of the hand through the markers, and the user's hand may be detected based on the information obtained. When using such a sensor capable of measuring three-dimensional position, for example, a sensor capable of measuring three-dimensional position may be provided in the HMD 100 instead of or in addition to the hand detection image capture unit 110 shown in FIG. 1, and the measurement results of the sensor may be input to the detection unit 210. Also, while the embodiment has been described in which a sensor capable of measuring three-dimensional position is provided in the HMD 100, the sensor may be provided separately from the HMD 100 and the information processing device 200, and the measurement results of the sensor may be input to the detection unit 210. Note that when a sensor is used to measure the three-dimensional position of the hand joint points, the reference coordinate system is the sensor coordinate system defined by the sensor capable of measuring three-dimensional position. Therefore, the determination method by determination unit 220 in step S103 in FIG. 4 differs depending on whether a hand detection image captured by hand detection image capturing unit 110 is used or a sensor capable of measuring three-dimensional positions is used.

[0032] 4, the determination unit 220 determines whether or not the user's hand has been detected by the detection unit 210. Here, the determination unit 220 may determine that the hand has been detected if the three-dimensional position of at least one of the above-mentioned joint points of the hand in real space can be determined, or may determine that the hand cannot be detected if the three-dimensional positions of all of the joint points of the hand cannot be determined.

[0033] In step S102 of FIG. 4, if the determination unit 220 determines that the detection unit 210 has detected the user's hand (S102 / Yes), the process proceeds to step S103. When proceeding to step S103 in FIG. 4, the determination unit 220 determines whether the hand of the experiencer detected by the detection unit 210 is within the visual field range of the experiencer (in this embodiment, shown in the presentation image captured by the presentation image capturing unit 120).

[0034] Here, a specific example in step S103 of FIG. 4 will be described. In this embodiment, in step S103 of FIG. 4, as described above, the determination unit 220 determines whether the hand of the experiencer detected by the detection unit 210 is shown in the presentation image captured by the presentation image capturing unit 120. In the case of this embodiment, the determination unit 220 first converts the position of the hand in the coordinate system based on the hand detection image capturing unit 110 to the position in the coordinate system based on the presentation image capturing unit 120. Then, the determination unit 220 performs perspective projection transformation using the internal parameters related to the presentation image capturing unit 120 to convert the position in the coordinate system based on the presentation image capturing unit 120 to the position in the coordinate system of the image. And the determination unit 220 determines whether the hand of the experiencer is shown in the presentation image captured by the presentation image capturing unit 120 according to whether the position of the hand in the coordinate system of this image is within the presentation image of the presentation image capturing unit 120. Note that it may be determined that the hand of the experiencer is shown in the presentation image when the hand of the experiencer is shown in either the left image or the right image of the presentation image capturing unit 120, or it may also be determined that the hand of the experiencer is shown in the presentation image when the hand of the experiencer is shown in both images.

[0035] Furthermore, when a sensor capable of measuring the above-described three-dimensional position is provided and the sensor is used to determine whether the user's hand detected by the detection unit 210 appears in the presented image captured by the presented image capture unit 120, the following configuration may be adopted. In this configuration, the positions of the hand joint points detected based on the coordinate system of the sensor capable of measuring the three-dimensional position must be converted to positions in a coordinate system based on the presented image capture unit 120. Therefore, for example, a marker for measuring the three-dimensional position may be attached to the HMD 100, and the relative positional relationship between the presented image capture unit 120 and the marker may be maintained. In this case, the determination unit 220 may convert the hand position detected in the sensor's coordinate system to a position based on the coordinate system of a marker attached to the HMD 100 that is also detected in the sensor's coordinate system, and further convert the position to a coordinate system based on the presented image capture unit 120. The technology for using a sensor capable of measuring such three-dimensional position to convert the position of the detected object (hand) into the coordinate system of the presentation image capture unit 120 provided in the HMD 100 worn by the user can use any known technology, and technologies other than the methods described above may also be used.

[0036] 4, if the determination unit 220 determines that the user's hand detected by the detection unit 210 is not within the field of view of the user (S103 / No), the process proceeds to step S104. That is, the process proceeds to step S104 in FIG. 4 when the user's hand is detected by the detection unit 210 (S102 / Yes) and the detected user's hand is not within the field of view of the user (S103 / No), which corresponds to a predetermined state. 4, the notification unit 230 notifies the HMD 100 of information indicating that the experiencer's hand detected by the detection unit 210 is not within the field of view of the experiencer (not shown in the presented image) as a predetermined state, as the hand detection status. Specifically, in this embodiment, the display control means 231 including the notification unit 230 controls the display unit 130 to display information indicating that the experiencer's hand detected by the detection unit 210 is not within the field of view of the experiencer (not shown in the presented image) as a predetermined state, as the hand detection status. More specifically, in this embodiment, the notification unit 230 generates an image indicating to the experiencer that their hand has been detected. Furthermore, the notification unit 230 generates a mixed reality space image for the right eye and a mixed reality space image for the left eye as experiential images by combining the generated image with images from the right imaging unit and the left imaging unit of the presented image capturing unit 120. Then, the display control means 231 including the notification unit 230 transmits the generated mixed reality space image for the right eye and the mixed reality space image for the left eye to the right eye display and the left eye display of the display unit 130, respectively, and displays them.

[0037] FIG. 5 is a diagram for explaining an example of the process in step S104 of FIG. FIG. 5(a) shows a hand detection image 510 captured by the hand detection image capturing unit 110, and this hand detection image 510 includes the user's hand 511. In this case, the detection unit 210 can detect the user's hand 511 from the hand detection image 510 shown in FIG. 5(a), and can further detect the user's hand 511 by recognizing the joint points of the user's hand 511. FIG. 5(b) shows a presentation image 520 captured by the presentation image capturing unit 120, and the user's hand 511 shown in FIG. 5(a) is not shown in this presentation image 520. Therefore, the display control means 231, which includes the notification unit 230, controls the display unit 130 to display an image 530 as information indicating that the user's hand 511 detected by the detection unit 210 is not within the user's field of view (not shown in the presentation image 520), which is a predetermined state. In this case, the display control means 231 including the notification unit 230 controls the display unit 130 to display an experience image obtained by superimposing an image 530 indicating that the hand has been detected on the presented image 520. In this case, as shown in FIG. 5(b), the image 530 indicating that the hand has been detected may be, for example, an image indicating joint points 531 of the user's hand 511 detected by the detection unit 210. As shown in FIG. 5(b), by displaying the image 530 indicating the joint points 531 of the user's hand 511 detected by the detection unit 210, the user can easily understand how their hand has been detected. Note that in this embodiment, the display control means 231 including the notification unit 230 may control the display unit 130 to display text information indicating that the user's hand 511 has been detected, as information indicating the above-mentioned predetermined state.

[0038] Here, we return to the explanation of FIG. In step S102 of FIG. 4, if the determination unit 220 determines that the detection unit 210 has not detected the user's hand (S102 / No), the process proceeds to step S105. When the process proceeds to step S105 in FIG. 4 , the notification unit 230 notifies the HMD 100 of information indicating that the user's hand has not been detected as the hand detection status. Specifically, in this embodiment, the display control means 231 including the notification unit 230 controls the display unit 130 to display information indicating that the user's hand has not been detected as the hand detection status. More specifically, in this embodiment, the notification unit 230 generates an image indicating to the user that their hand has not been detected. Furthermore, the notification unit 230 combines the generated image with the image from the right imaging unit and the image from the left imaging unit of the presentation image capture unit 120 to generate a mixed reality space image for the right eye and a mixed reality space image for the left eye as experience images. Thereafter, the display control means 231 including the notification unit 230 transmits the generated mixed reality space image for the right eye and the mixed reality space image for the left eye to the right eye display and the left eye display of the display unit 130, respectively, to display them. This allows the user to understand that their hand has not been detected.

[0039] When the process of step S104 in Fig. 4 is completed, or when the process of step S105 in Fig. 4 is completed, the process proceeds to step S106. Also, when the determination unit 220 determines in step S103 in Fig. 4 that the experiencer's hand detected by the detection unit 210 is within the field of view of the experiencer (S103 / Yes), the process proceeds to step S106. 6, processor 310 of information processing device 200 determines whether or not the experiencer has given an instruction to end the experience. If processor 310 determines in step S106 of FIG. 6 that the experiencer has not given an instruction to end the experience (S106 / No), the process returns to step S101 and repeats the processes from step S101 onward.

[0040] Furthermore, in step S106 of FIG. 6, if processor 310 determines that the experiencer has instructed to end the experience (S106 / Yes), the processing of the flowchart shown in FIG. 4 ends.

[0041] The information processing device 200 according to the first embodiment described above is configured to be able to communicate with the HMD 100, which is a display device including a display unit 130 that is positioned to cover the user's eyes and displays an experiential image based on real space to the user, and has the following configuration. Specifically, the information processing device 200 according to the first embodiment includes a detection unit 210 that is a detection means for detecting the user's hand based on information obtained by capturing or measuring an area wider than the user's field of view in real space. Furthermore, the information processing device 200 according to the first embodiment includes a display control unit 231 that controls the display unit 130 to display information indicating the predetermined state when the detection unit 210 detects the user's hand and the detected user's hand is not within the user's field of view. Specifically, in the first embodiment, the field of view of the user in real space is the range of the experiential image displayed on the display unit 130. The detection unit 210 detects the user's hand based on information obtained by capturing or measuring an area wider than the range of the experiential image displayed on the display unit 130. The display control unit 231 controls the display of the experiential image to include the information indicating the predetermined state. With this configuration, when the user's hand is detected in an area that is not within the user's field of vision, the user can understand the detection status of their own hand (how their hand is detected). As a result, the user can more easily operate the CG (including UI operations) using their own hand, and can enjoy a more comfortable experience.

[0042] <Modification 1 of the First Embodiment> The first embodiment described above is a form in which, in a video see-through HMD 100 equipped with presentation image capture unit 120, if a user's hand is detected even in an area where the user cannot see the user's hand, the hand detection status is displayed in the experiential image seen by the user. In Variation 1 of the first embodiment, in an optical see-through HMD 100, if a user's hand is detected even in an area where the user cannot see the user's hand, the hand detection status is displayed in the experiential image seen by the user.

[0043] In the following description of Modification 1 of the first embodiment, matters common to the first embodiment described above will be omitted, and only matters different from the first embodiment described above will be described.

[0044] In the first modification of the first embodiment, in the schematic configuration of the information processing system 10-1 shown in FIG. 1, the internal configuration of the HMD 100 and the processing content of the determination unit 220 of the information processing device 200 are different from those of the first embodiment described above.

[0045] The optical see-through HMD 100 according to the first modification does not need to include the presentation image capturing unit 120 shown in Fig. 1. In this case, the determination unit 220 of the information processing device 200 does not receive the presentation image from the presentation image capturing unit 120.

[0046] In the first embodiment described above, the judgment unit 220 determines whether the experiencer's hand, detected by the detection unit 210, is reflected in the presentation image captured by the presentation image capturing unit 120, thereby determining whether the experiencer's hand is within the field of view of the experiencer.

[0047] In contrast, the determination unit 220 in Modification 1 defines the field of view of the experience as a setting and determines whether the user's hand detected by the detection unit 210 is within the field of view of the experience. In Modification 1, to define the field of view of the experience, the user's eyes are considered to be virtual imaging units. In Modification 1, the relative relationship between the user's eyes and the hand detection image capture unit 110 and the range the user can see are defined as the angle of view. In Modification 1, additional internal parameters of the capture unit (focal length, principal point) are provisionally defined and saved as settings in the determination unit 220. The determination unit 220 in Modification 1 considers the user's eyes as virtual imaging units and determines whether the user's hand is within the field of view of the experience using the internal parameters set above and the relative relationship between the user's eyes and the hand detection image capture unit 110. This allows the user to understand the detection status of their own hand (how their hand is detected) even in the optical see-through HMD 100 when their hand is detected in an area outside the field of view of the experience.

[0048] <Modification 2 of the First Embodiment> In the first embodiment described above, if the user's hand is detected even in an area where the user cannot see the user's hand, the hand detection status is displayed in the experience image seen by the user. In the second modification of the first embodiment, the detection unit 210 has an identification function for distinguishing between the user's hand and the hand of another person who is not the user, and is configured to not include the hand of another person as a detection target.

[0049] In the following description of Modification 2 of the first embodiment, matters common to the first embodiment described above will be omitted, and only matters different from the first embodiment described above will be described.

[0050] In the second modification of the first embodiment, the processing content of the detection unit 210 of the information processing device 200 is different from that of the first embodiment described above.

[0051] The detection unit 210 in the first embodiment described above detects hands from hand detection images captured by the hand detection image capturing unit 110. In this case, there is a possibility that the hand of another person may be detected, and the user may be notified of the result that the hand of another person has been detected.

[0052] Therefore, in the present modified example 2, the detection unit 210 detects a hand from the hand detection image captured by the hand detection image capturing unit 110, and then determines whether the detected hand is someone else's hand. For example, the detection unit 210 determines that the hand is someone else's hand if the position of the hand detected from the hand detection image captured by the hand detection image capturing unit 110 is away from the hand detection image capturing unit 110 of the HMD 100 by a threshold or more. In this case, for example, the detection unit 210 determines that the hand is the user's hand if the position of the hand detected from the hand detection image captured by the hand detection image capturing unit 110 is not away from the hand detection image capturing unit 110 of the HMD 100 by a threshold or more.

[0053] In addition, in the present modified example 2, when the color of the hand on the hand detection image detected from the hand detection image captured by the hand detection image capturing unit 110 differs from the color of the experiencer's hand registered in advance, the detection unit 210 determines that the hand is the hand of another person. That is, the detection unit 210 distinguishes between the experiencer's hand and the hand of another person by color.

[0054] Furthermore, in the present modified example 2, when a state where a hand has not been detected changes to a state where a hand is detected for the first time, and the position of the detected hand on the image is in the upward direction of the image (HMD 100), the detection unit 210 determines that the hand is that of another person. This is because, when the user wears the HMD 100, it is considered that the user's hand is unlikely to come in from the upward direction of the image (HMD 100).

[0055] As in the second modification of the first embodiment described above, when the detection unit 210 determines that the detected hand is the hand of another person, it determines that the user's hand has not been detected. According to the second modification of the first embodiment, when notifying the user of the detection status of his / her own hand, if the hand of another person is detected, it is possible to notify the user that the user's hand has not been detected. This allows the user to recognize that his / her own hand has been detected more accurately.

[0056] (Second embodiment) Next, a second embodiment will be described. In the following description of the second embodiment, matters common to the first embodiment will be omitted, and only matters different from the first embodiment will be described.

[0057] In the first embodiment and the first modification of the first embodiment described above, when the user's hand is detected in an area where the user cannot see the hand, the hand detection status is displayed in the experiential image seen by the user. In contrast, in the second embodiment, the user is notified of the hand detection status and is also notified when the user's hand is about to move into an area where it cannot be detected.

[0058] Fig. 6 is a diagram showing an example of a schematic configuration of an information processing system 10-2 according to the second embodiment. In Fig. 6, the same components as those shown in Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0059] The information processing device 200 according to the second embodiment shown in FIG. 6 has a functional configuration of a detection unit 210, a calculation unit 610, and a display control means 231 including a notification unit 230.

[0060] The calculation unit 610 is a calculation means that calculates the position of the user's hand detected by the detection unit 210 in the presentation image captured by the presentation image capturing unit 120 and the hand detection image captured by the hand detection image capturing unit 110.

[0061] Fig. 7 is a flowchart showing an example of a processing procedure in a control method for the information processing device 200 according to the second embodiment. In Fig. 7, the same processing steps as those in the flowchart shown in Fig. 4 are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0062] When the processing of the flowchart in FIG. 7 starts, first, the same processing as steps S101 and S102 in FIG. 4 is performed.

[0063] In step S102 of FIG. 7, if the determination unit 220 determines that the detection unit 210 has detected the user's hand (S102 / Yes), the process proceeds to step S201. 7, the calculation unit 610 calculates the position of the user's hand detected by the detection unit 210 in the presentation image captured by the presentation image capturing unit 120 and the hand detection image captured by the hand detection image capturing unit 110. The position of the user's hand detected by the detection unit 210 in each image can be calculated by performing the coordinate transformation and perspective projection transformation described in the processing of step S103 in FIG.

[0064] 7, the notification unit 230 generates information (image) indicating the detection result of the user's hand as the hand detection status in accordance with the calculation result of the calculation unit 610 in step S201, and notifies the HMD 100. Specifically, in this embodiment, the display control means 231 including the notification unit 230 controls the display unit 130 to display information (image) indicating the detection result of the user's hand as the hand detection status in accordance with the calculation result of the calculation unit 610 in step S201.

[0065] FIG. 8 is a diagram for explaining an example of the process in step S202 of FIG. Fig. 8(a) is a diagram showing an example of the positional relationship between a hand detection image 810 captured by the hand detection image capturing section 110 and a presentation image 820 captured by the presentation image capturing section 120 in real space. Fig. 8(a) also shows a first region 811 and a second region 812 of the hand detection image 810, and a first region 821 and a second region 822 of the presentation image 820. Fig. 8(b) is a diagram showing each of the regions (811, 812, 821, 822) shown in Fig. 8(a), the display colors of the joint points when the user's hand is in each of the regions, and an explanation of each of the regions.

[0066] In FIG. 8(a), when the user's hand is in the first region 821 of the presentation image 820, the user can see the hand, but since the hand is close to the edge of the hand detection image 810, which the user cannot see, the user may soon be unable to detect (see) the hand. Therefore, the display control means 231 including the notification unit 230 displays the color of the hand joint point 531 in the image 530 of FIG. 5 in yellow based on FIG. 8(b), thereby notifying the user that the user's hand will soon be unable to detect (see) the hand in the region visible to the user. In this way, in the second embodiment, even if the user is moving their hand in a region that is visible, the user can be notified that the hand is moving to a region where it will likely become undetectable (unseen). Therefore, it is easy to understand how the user should move their hand to make it undetectable (unseen).

[0067] 8(a), if the user's hand is in the second area 822 of the presented image 820, there is a low possibility that the hand will become undetectable (invisible) in the area visible to the user. Therefore, in the second embodiment, the display control means 231 including the notification unit 230 displays the color of the hand joint points 531 in the image 530 of FIG. 5 in blue based on FIG. 8(b), thereby notifying the user that it is okay to move the hand widely in the area visible to the user.

[0068] 8(a), if the user's hand is in the second region 812 of the hand detection image 810, the user cannot see his / her own hand, but there is a low possibility that the detection unit 210 will not be able to detect his / her own hand. Therefore, the display control means 231 including the notification unit 230 displays the color of the hand joint points 531 in the image 530 of FIG. 5 in green based on FIG. 8(b), thereby notifying the user that the detection unit 210 can detect the hand even if the user moves the hand widely, even in a region that is not visible to the user.

[0069] 8(a), if the user's hand is in the first region 811 of the hand detection image 810, the user cannot see his / her own hand, and it is highly likely that the detection unit 210 will soon be unable to detect his / her own hand. Therefore, in the second embodiment, the display control means 231 including the notification unit 230 displays the color of the hand joint points 531 in red in the image 530 of FIG. 5 based on FIG. 8(b), to notify the user that the detection unit 210 will likely be unable to detect his / her hand.

[0070] In the information processing device 200 according to the second embodiment, the display control means 231 changes the display form of the information indicating the above-described predetermined state (in this embodiment, the display color of the hand joint point 531) according to the position of the user's hand detected by the detection unit 210. With this configuration, the user can understand how to move his or her hand so that it becomes undetectable.

[0071] (Third embodiment) Next, a third embodiment will be described. In the following description of the third embodiment, matters common to the first and second embodiments will be omitted, and only matters different from the first and second embodiments will be described.

[0072] In the first embodiment described above, when a predetermined state occurs in which a hand is detected by the detection unit 210 in an area not visible to the user, information indicating the predetermined state is displayed on the display unit 130 to notify the user. In the third embodiment, the user is notified only when he or she desires to be notified of the hand detection result.

[0073] Fig. 9 is a diagram showing an example of a schematic configuration of an information processing system 10-3 according to the third embodiment and its modified example 1. In Fig. 9, the same components as those shown in Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0074] 9 has a functional configuration of a detection unit 210, a determination unit 220, a notification unit 230, a storage unit 910, a calculation unit 920, and a generation unit 930. Here, in the third embodiment, a display control means 931 including the notification unit 230 and the generation unit 930 is configured.

[0075] The storage unit 910 stores computer graphics (CG) information. The CG information stored in the storage unit 910 may include information on an object to be manipulated, such as pinched, using position information on the joint points 531 of the user's hand detected by the detection unit 210. The CG information stored in the storage unit 910 may also include information on a user interface. In this case, the display position of the CG may be changed by the user interface.

[0076] The determination unit 220 acquires CG information from the storage unit 910 and determines whether the acquired CG information includes information on CG that can be operated with hands. As a result, when the determination unit 220 determines that there is information on CG that can be operated with hands, the notification unit 230 may determine that operation using hands may be necessary, and may notify the user of the hand detection result by the detection unit 210. Furthermore, when the determination unit 220 determines that there is no information on CG that can be operated with hands, the notification unit 230 may determine that operation using hands is unnecessary, and may not notify the user of the hand detection result by the detection unit 210.

[0077] Calculation unit 920 calculates the position and orientation of hand detection image capturing unit 110 using the hand detection image captured by hand detection image capturing unit 110 .

[0078] The generation unit 930 generates an image of the mixed reality space based on the outputs from the notification unit 230, the storage unit 910, and the calculation unit 920, and outputs this to the display unit 130 as an experience image.

[0079] The display control means 931, which includes the notification unit 230 and the generation unit 930, controls the display of information indicating the hand detection results by the detection unit 210 on the display unit 130 to notify the user when the determination unit 220 determines that there is CG information that can be operated with the hands.

[0080] Fig. 10 is a flowchart showing an example of a processing procedure in a control method for the information processing device 200 according to the third embodiment. In Fig. 10, the same processing steps as those in the flowchart shown in Fig. 4 are assigned the same step numbers, and detailed descriptions thereof will be omitted.

[0081] When the processing of the flowchart in FIG. 10 starts, first, the same processing as steps S101 to S103 in FIG. 4 is performed.

[0082] 10, if the determination unit 220 determines that the user's hand detected by the detection unit 210 is not within the field of view of the user (S103 / No), the process proceeds to step S301. That is, the process proceeds to step S301 in Fig. 10 when the user's hand is detected by the detection unit 210 (S102 / Yes) and the detected user's hand is not within the field of view of the user (S103 / No), which corresponds to a predetermined state. When proceeding to step S301 in FIG. 10, the determination unit 220 acquires CG information from the storage unit 910, and determines whether the acquired CG information includes information on a CG (virtual object) that can be manipulated by hand.

[0083] In step S301 of FIG. 10, if the determination unit 220 determines that the acquired CG information includes information about a CG (virtual object) that can be manipulated by hand (S301 / Yes), the process proceeds to step S302. When the process proceeds to step S302 in FIG. 10, the notification unit 230 generates an image as information indicating the hand detection result by the detection unit 210, which image indicates to the user that the hand has been detected.

[0084] When the process of step S302 in Fig. 10 is completed, or when the process of step S105 in Fig. 10 is completed, the process proceeds to step S303. Also, when the determination unit 220 determines in step S103 in Fig. 10 that the experiencer's hand detected by the detection unit 210 is within the field of view of the experiencer (S103 / Yes), the process proceeds to step S303. Proceeding to step S303 of FIG. 10 , first, the calculation unit 920 calculates the position and orientation of the hand detection image capturing unit 110 via the detection unit 210 using the hand detection image captured by the hand detection image capturing unit 110. Specifically, first, the calculation unit 920 detects image coordinates of feature points in the hand detection image captured by the hand detection image capturing unit 110. Here, the process of extracting feature points from an image may involve determining points where the brightness gradient between neighboring pixels is equal to or greater than a threshold value as feature points. In this case, the brightness gradient is the amount of change in density between adjacent pixels on the image. Furthermore, the brightness gradient is calculated by applying, for example, a Sobel operator or a Prewitt operator in the horizontal and vertical directions of the image. Then, the calculation unit 920 extracts feature points using the output values ​​as the brightness gradient. Next, the calculation unit 920 performs SLAM processing based on the extracted feature points to calculate and estimate the position and orientation of the hand detection image capturing unit 110. Alternatively, the HMD 100 may be provided with a sensor for detecting the position and orientation of the hand detection image capture unit 110, or the position and orientation of the hand detection image capture unit 110 may be estimated by combining the optical sensor, magnetic sensor, and ultrasonic sensor described above. Next, the calculation unit 920 estimates the position and orientation of the hand detection image capture unit 110, and then calculates the positions and orientations of the left and right capture units of the presentation image capture unit 120 using the position and orientation relative to the presentation image capture unit 120 that was calculated in advance. Thereafter, the generation unit 930 first uses the CG information acquired from the holding unit 910 to construct a virtual reality space in which each CG is arranged. Then, the generation unit 930 uses the internal parameters of the presentation image capture unit 120 to generate an image of the virtual reality space as seen from the positions and orientations of the right and left capture units of the presentation image capture unit 120 calculated by the calculation unit 920. Next, the generation unit 930 synthesizes the generated image of the virtual reality space with the images of the real space acquired from the right and left imaging units of the presentation image capture unit 120 to generate an image of the mixed reality space that serves as an experiential image for the user to experience. In this case, the image synthesis process is performed by overlaying the image of the virtual reality space on the image of the real space. In other words, pixels outside the CG area become a composite image in which pixels of the real space are displayed. Here, the user may experience only the image of the virtual reality space as the experiential image.In this case, generation section 930 does not need to synthesize the presentation image captured by presentation image capturing section 120 with the image of the virtual reality space.

[0085] Next, in step S304 of FIG. 10 , if an image indicating to the user that the user's hand has been detected is generated in step S302, the generation unit 930 performs processing to synthesize the image into the experience image generated in step S303. For example, the generation unit 930 performs processing to synthesize an image indicating to the user that the user's hand has been detected into the upper right corner of the experience image. Note that in this step, if information (image) indicating that the user's hand has not been detected is generated in step S105, the generation unit 930 may perform processing to synthesize the information (image) into the experience image generated in step S303. Then, the display control means 931, which includes the notification unit 230 and the generation unit 930, controls the display of the experience image generated in this step on the display unit 130 to notify the user. This allows the user wearing the HMD 100 to experience a mixed reality space (or a virtual reality space). Furthermore, in a situation where it is necessary to check whether the user's hand has been detected, the user can experience a mixed reality space or the like while understanding the hand detection status.

[0086] In the information processing device 200 according to the third embodiment, when the above-mentioned predetermined state is reached and there is CG that can be operated by the user's hands (S301 / Yes), the display control means 931 controls the display unit 130 to display information indicating that the predetermined state is reached. According to this configuration, the hand detection status is notified when it is considered that the user needs to operate the CG (virtual object) with their hands, so it is possible to prevent the hand detection status from being notified when the user does not need to operate the CG (virtual object) with their hands.

[0087] <Modification 1 of the third embodiment> In the third embodiment described above, the user is notified of the hand detection status when it is deemed necessary for the user to operate a CG (virtual object) with their hand. Variation 1 of the third embodiment is a mode in which the hand detection status is displayed in a position that is easy for the user to see.

[0088] In the following description of Modification 1 of the third embodiment, matters common to the above-described third embodiment will be omitted, and only matters different from the above-described first embodiment will be described.

[0089] In Modification 1 of the third embodiment, the processing content of the generation unit 930 of the information processing device 200 differs from that of the third embodiment described above. Specifically, in the third embodiment, the generation unit 930 performs processing to synthesize and include an image indicating to the user that the hand has been detected at a fixed position, such as the upper right corner, of the experience image, allowing the user to experience the experience. However, the position at which the image indicating to the user that the hand has been detected is synthesized with the experience image does not have to be a fixed position.

[0090] In this first modification, for example, as shown in FIG. 9 , the HMD 100 may include an observation position detection unit 140 that detects the position at which the user is observing the image of the mixed reality space displayed on the display unit 130. The HMD 100 may include the observation position detection unit 140 that captures the user's eyes, and the observation position detection unit 140 detects the observation position of the display unit 130 based on the direction of the user's eyes using known technology. Therefore, detailed description of the observation position detection unit 140 will be omitted. In this first modification, the display control unit 931, including the notification unit 230 and the generation unit 930, controls the display of information indicating the above-described predetermined state by combining it with the experience image near the observation position of the display unit 130 detected by the observation position detection unit 140. As a result, for example, when a CG (virtual object) at the user's observation position is set as an operation target, the detection status of the user's hand, which is an input device for operating the object, is displayed near the CG. This allows the user to simultaneously view the CG and the detection status of the user's hand without frequently moving their line of sight. This allows the user to concentrate on manipulating the CG (virtual object) with their own hands.

[0091] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. This program and a computer-readable storage medium storing the program are included in the present invention.

[0092] It should be noted that the above-described embodiments of the present invention are merely illustrative examples of the implementation of the present invention, and the technical scope of the present invention should not be construed as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0093] The disclosure of this embodiment includes the following configuration, method, and program. [Configuration 1] An information processing device configured to be able to communicate with a display device including a display unit that is arranged to cover the eyes of an experiencer and displays an experience image based on real space to the experiencer, a detection means for detecting the hands of the experiencer based on information obtained by capturing or measuring an image of a range in the real space that is wider than the visual field range of the experiencer; a display control means for controlling the display of information indicating a predetermined state when the user's hand is detected by the detection means and the detected user's hand is not within the user's visual field range; and An information processing device comprising: [Configuration 2] a visual field range of the experience in the real space is a range of the experience image displayed on the display unit, the detection means detects the hands of the user based on information obtained by capturing or measuring an area wider than the area of ​​the experience image displayed on the display unit; The display control means controls the display of the experience image so that information indicating the predetermined state is included in the experience image. 2. The information processing device according to configuration 1, [Configuration 3] The detection means has a discrimination function for discriminating between the hands of the person experiencing the experience and the hands of other people who are not the person experiencing the experience. 3. The information processing device according to configuration 1 or 2. [Configuration 4] The discrimination function of the detection means discriminates between the user's hand and the other person's hand by color. 4. The information processing device according to configuration 3. [Configuration 5] The identification function of the detection means determines that the detected hand is the hand of the other person when the detected hand is farther away from the display device than a threshold value, and determines that the detected hand is the hand of the user when the detected hand is not farther away from the display device than the threshold value. 4. The information processing device according to configuration 3. [Configuration 6] The identification function of the detection means determines that the hand is the hand of another person when the detected hand is detected in the upward direction of the display device when the state where the hand is not detected changes to the state where the hand is detected. 4. The information processing device according to configuration 3. [Configuration 7] The display control means changes the display form of the information indicating the predetermined state in accordance with the position of the user's hand detected by the detection means. 7. The information processing device according to any one of configurations 1 to 6. [Configuration 8] The change in the display form is a change in the display color. 8. The information processing device according to configuration 7. [Configuration 9] The display control means controls the display of information indicating that the predetermined state is being achieved on the display unit when a CG that can be operated by the user's hands is present in the predetermined state. 9. The information processing device according to any one of configurations 1 to 8. [Configuration 10] The display control means controls the display of information indicating the predetermined state in the vicinity of an observation position of the display unit based on an eye direction of the user. 10. The information processing device according to any one of configurations 1 to 9. [Configuration 11] The display control means controls the display of an image showing the joint points of the user's hand detected by the detection means on the display unit as information showing the predetermined state. 11. The information processing device according to any one of configurations 1 to 10. [Configuration 12] The display control means controls the display unit to display text information indicating that the user's hand has been detected as information indicating the predetermined state. 11. The information processing device according to any one of configurations 1 to 10. [Method 1] A control method for an information processing device configured to be able to communicate with a display device including a display unit that is arranged to cover the eyes of an experiencer and displays an experiential image based on real space to the experiencer, a detection step of detecting the hands of the experiencer based on information obtained by capturing or measuring an image of a range in the real space that is wider than the field of view of the experiencer; a display control step of controlling the display unit to display information indicating a predetermined state when the user's hand is detected by the detection step and the detected user's hand is not within the user's visual field; and 1. A method for controlling an information processing device, comprising: [Program 1] 13. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0094] 10: Information processing system, 100: HMD, 110: Hand detection image capturing unit, 120: Presentation image capturing unit, 130: Display unit, 200: Information processing device, 210: Detection unit, 220: Determination unit, 230: Notification unit, 231: Display control means

Claims

1. An information processing device configured to be able to communicate with a display device including a display unit that is arranged to cover the eyes of an experiencer and displays an experience image based on real space to the experiencer, a detection means for detecting the hands of the experiencer based on information obtained by capturing or measuring an image of a range in the real space that is wider than the visual field range of the experiencer; a display control means for controlling the display of information indicating a predetermined state when the user's hand is detected by the detection means and the detected user's hand is not within the user's visual field range; and An information processing device comprising:

2. a visual field range of the experience in the real space is a range of the experience image displayed on the display unit, the detection means detects the hands of the user based on information obtained by capturing or measuring an area wider than the area of ​​the experience image displayed on the display unit; The display control means controls the display of the experience image so that information indicating the predetermined state is included in the experience image.

2. The information processing apparatus according to claim 1, wherein:

3. The detection means has a discrimination function for discriminating between the hands of the person experiencing the experience and the hands of other people who are not the person experiencing the experience.

2. The information processing apparatus according to claim 1, wherein:

4. The discrimination function of the detection means discriminates between the user's hand and the other person's hand by color.

4. The information processing apparatus according to claim 3,

5. The identification function of the detection means determines that the detected hand is the hand of the other person when the detected hand is farther away from the display device than a threshold value, and determines that the detected hand is the hand of the user when the detected hand is not farther away from the display device than the threshold value.

4. The information processing apparatus according to claim 3,

6. The identification function of the detection means determines that the hand is the hand of another person when the detected hand is detected in the upward direction of the display device when the state where the hand is not detected changes to the state where the hand is detected.

4. The information processing apparatus according to claim 3,

7. The display control means changes the display form of the information indicating the predetermined state in accordance with the position of the user's hand detected by the detection means.

2. The information processing apparatus according to claim 1, wherein:

8. The change in the display form is a change in the display color.

8. The information processing apparatus according to claim 7,

9. The display control means controls the display of information indicating that the predetermined state is occurring on the display unit when CG that can be operated by the user's hands is present in the predetermined state.

2. The information processing apparatus according to claim 1, wherein:

10. The display control means controls the display of information indicating the predetermined state in the vicinity of an observation position of the display unit based on an eye direction of the user.

2. The information processing apparatus according to claim 1, wherein:

11. The display control means controls the display of an image showing the joint points of the user's hand detected by the detection means on the display unit as information showing the predetermined state.

2. The information processing apparatus according to claim 1, wherein:

12. The display control means controls the display unit to display text information indicating that the user's hand has been detected as information indicating the predetermined state.

2. The information processing apparatus according to claim 1, wherein:

13. A control method for an information processing device configured to be able to communicate with a display device including a display unit that is arranged to cover the eyes of an experiencer and displays an experiential image based on real space to the experiencer, a detection step of detecting the hands of the experiencer based on information obtained by capturing or measuring an image of a range in the real space that is wider than the field of view of the experiencer; a display control step of controlling the display unit to display information indicating a predetermined state when the user's hand is detected by the detection step and the detected user's hand is not within the user's visual field; and 1. A method for controlling an information processing device, comprising:

14. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 12.

Images