Information processing device, user guide presentation method, and head-mounted display

Abstract

To solve the problem that it is difficult for a user wearing an HMD to recognize a state of a real world.SOLUTION: As a user guide, a state where a real space including an imaging apparatus and a user is overlooked is represented by an image (a) based on a position of the imaging apparatus or an image (b) based on an orientation of the user. In both of the images, a rectangle 130 indicating the imaging apparatus and a circle 132 indicating the user are represented so as to reflect actual physical relation and orientations. Further, an area 134 indicating a play area in a horizontal direction, which is determined by a field angle of a camera and so on and an arrow 136 indicating a direction in which the user should move are represented.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing device that processes information based on a captured image, a user guide presentation method performed by the information processing device, and a head-mounted display that displays images. [Background technology]

[0002] Players wear a head-mounted display (hereinafter referred to as "HMD") connected to a game console and play games while looking at the displayed screen (see, for example, Patent Document 1). For example, by acquiring the position and posture of the user's head and displaying an image of the virtual world so that the field of view changes according to the direction of the user's face, it is possible to create a situation in which the user feels as if they are immersed in the virtual world. The position and posture of the user are generally acquired based on the analysis results of visible light or infrared images captured of the user, or measurements from a motion sensor built into the HMD. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5580855 specification Summary of the Invention [Problem to be solved by the invention]

[0004] Technologies that perform some kind of information processing based on captured images assume that the user or other subject is within the camera's field of view. Furthermore, depending on the content of the information processing, the ideal area in which the user should be located may be further limited. However, since users cannot see the outside world while wearing an HMD, they may lose their sense of direction or become so engrossed in the game that they end up moving to an unexpected location in real space. This can lead to problems such as a deterioration in the accuracy of information processing, a breakdown, or collisions with other objects, but the cause is difficult to identify unless the HMD is removed. If such situations occur frequently, the view of the virtual world created using an HMD will be impaired.

[0005] The present invention has been made in consideration of these problems, and its purpose is to provide a technology that allows users to continue enjoying the world displayed by an HMD with minimal burden. [Means for solving the problem]

[0006] One aspect of the present invention relates to an information processing device that includes a position information acquisition unit that acquires position information of a head-mounted display, an information processing unit that processes information using the position information, an output data generation unit that generates and outputs data of a display image of a virtual world to be displayed as a result of the information processing, and a user guide generation unit that generates data of a user guide image that semi-transparently represents the state of the real world in which the user is located, and is characterized in that the output data generation unit superimposes the user guide image on the display image of the virtual world while a condition for displaying the user guide is satisfied.

[0007] Another aspect of the present invention relates to a user guide presentation method, the user guide presentation method including the steps of: acquiring position information of a head-mounted display; performing information processing using the position information; generating and outputting data of a display image of a virtual world to be displayed as a result of the information processing; and generating data of a user guide image that semi-transparently represents the state of the real world where the user is located, by an information processing device, wherein the outputting step is characterized in that the user guide image is superimposed on the display image of the virtual world while a condition for displaying the user guide is satisfied.

[0008] Yet another aspect of the present invention relates to a head-mounted display that displays an image generated by an information processing device, and is characterized by including: a communication unit that transmits and receives data to and from the information processing device; and a display unit that displays a virtual world display image generated by the information processing device as a result of information processing performed by the information processing device using position information of the head-mounted display, and that displays a user guide image, which semi-transparently represents the state of the real world in which the user is located, superimposed on the virtual world display image while a condition for displaying the user guide is satisfied.

[0009] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, or recording medium on which a computer program is recorded, are also valid aspects of the present invention. [Effects of the Invention]

[0010] According to the present invention, a user wearing an HMD can continue to enjoy the world of expression with little burden. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system to which the present embodiment can be applied. [Figure 2] 1A to 1C are diagrams illustrating examples of the external shape of an HMD according to the present embodiment. [Figure 3] 1 is a diagram showing an internal circuit configuration of an information processing device according to an embodiment of the present invention; [Figure 4] 2 is a diagram showing the internal circuit configuration of the HMD according to the present embodiment. FIG. [Figure 5] FIG. 2 is a diagram showing a functional block configuration of the information processing device according to the present embodiment. [Figure 6] 10A and 10B are diagrams for explaining information that can be acquired from a captured image in the present embodiment. [Figure 7] 10A and 10B are diagrams illustrating examples of images when an overhead view is displayed as a user guide in the present embodiment. [Figure 8] 10 is a diagram illustrating an example of a display screen on which an image of a user guide is displayed in the present embodiment. FIG. [Figure 9] 10A and 10B are diagrams illustrating examples of images in which a side view is displayed as a user guide in the present embodiment. [Figure 10] FIG. 10 is a diagram showing another example of an image when an overhead view is displayed as a user guide in the present embodiment. [Figure 11] FIG. 10 is a diagram showing another example of an image when an overhead view is displayed as a user guide in the present embodiment. [Figure 12] 10A and 10B are diagrams illustrating examples of images in which a point cloud is displayed as a user guide in this embodiment. [Figure 13] FIG. 10 is a diagram showing another example of an image when an overhead view is displayed as a user guide in the present embodiment. [Figure 14] 10 is a flowchart showing a processing procedure in which the information processing device generates output data according to the user's movement in the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] 1 shows an example of the configuration of an information processing system to which this embodiment can be applied. The information processing system 8 includes an imaging device 12 that captures an image of an object, an information processing device 10 that processes information based on the captured image, a flat panel display 16 and an HMD 18 that display an image obtained as a result of the information processing, and an input device 14 that is operated by a user.

[0013] The information processing device 10, the imaging device 12, the input device 14, the flat display 16, and the HMD 18 may be connected by wired cables or by known wireless communication technology such as Bluetooth (registered trademark). Furthermore, depending on the information processing performed by the information processing device 10, the flat display 16 may not be required. Furthermore, the external shapes of these devices are not limited to those shown in the drawings. Furthermore, a device may be configured that integrates two or more of these devices. For example, the information processing device 10, the input device 14, and the flat display 16 may be realized as a mobile terminal equipped with them.

[0014] The imaging device 12 includes a camera that captures an image of a target such as a user at a predetermined frame rate, and a mechanism that generates output data of the captured image by performing general processing such as demosaic processing on the output signal from the camera and sends the generated output data to the information processing device 10. The camera includes a visible light sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor that is used in general digital cameras and digital video cameras. The imaging device 12 may include only one camera, or may be a so-called stereo camera in which two cameras are arranged left and right at a known distance as shown in the figure.

[0015] Alternatively, the imaging device 12 may be configured by combining a monocular camera with a device that irradiates an object with reference light such as infrared light and measures the reflected light. When a stereo camera or a mechanism for measuring reflected light is introduced, the position of the object can be determined in three-dimensional real space, which can further diversify the information processing by the information processing device 10 and the image display by the display device. Methods for determining the distance of the object from the camera based on the principle of triangulation using stereo images captured from left and right viewpoints by a stereo camera, and methods for determining the distance of the object from the camera based on TOF (Time of Flight) or pattern irradiation by measuring the reflection of reference light are both widely known.

[0016] Hereinafter, the description will be mainly focused on the case where the imaging device 12 captures stereo images, but as mentioned above, this is not intended to limit the present embodiment, and the imaging device 12 may include at least one camera. The information processing device 10 performs necessary information processing using data transmitted from the imaging device 12, and generates output data such as images and audio. The content of the processing performed by the information processing device 10 is not particularly limited, and may be determined as appropriate depending on the functions, applications, electronic content, etc. desired by the user.

[0017] For example, the information processing device 10 performs general face detection and tracking processing on the captured image to progress a game in which a character reflecting the user's movements as a target appears, or converts the user's movements into command inputs and performs information processing. At this time, markers provided on the input device 14 may be used to acquire the movements of the input device 14. Alternatively, by tracking multiple markers provided on the outer surface of the HMD 18, the position and posture of the head of the user wearing the HMD 18 may be identified, and the virtual world viewed from a viewpoint that moves in response to the position and posture may be displayed on the HMD 18. The output data generated by the information processing device 10 is transmitted to at least the HMD 18.

[0018] The HMD 18 is a display device worn by a user on the head, which displays images on a display panel such as an organic EL panel located in front of the user's eyes. For example, parallax images viewed from left and right viewpoints may be generated and displayed in the left and right areas obtained by dividing the display screen into two, thereby providing a stereoscopic view of the image. However, this is not intended to limit the scope of the present embodiment, and a single image may be displayed on the entire display screen. The HMD 18 may also have built-in speakers or earphones that output audio to positions corresponding to the user's ears.

[0019] The flat panel display 16 may be a television having a display for outputting two-dimensional images and a speaker for outputting sound, such as an LCD television, an OLED television, a plasma television, or a PC display. Alternatively, it may be the display and speaker of a tablet terminal or a mobile terminal. The input device 14 is operated by a user to receive requests such as those for starting and ending processing, selecting functions, and inputting various commands, and supplies these as electrical signals to the information processing device 10.

[0020] The input device 14 may be realized by any one of common input devices, such as a game controller, a keyboard, a mouse, a joystick, or a touchpad provided on the display screen of the flat panel display 16, or a combination thereof. The input device 14 may further include a light-emitting marker consisting of an element or a group of elements that emit light in a predetermined color. In this case, the information processing device 10 can track the movement of the marker using captured images, thereby making it possible to treat the movement of the input device 14 itself as a user operation. The input device 14 may also be configured only with a light-emitting marker and a mechanism for holding it.

[0021] 2 shows an example of the external shape of the HMD 18. In this example, the HMD 18 is made up of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that, when worn by the user, goes around the head and secures the device. The wearing band 106 is made of a material or has a structure that allows the length to be adjusted to fit the head circumference of each user. For example, it may be made of an elastic material such as rubber, or a buckle, gears, or the like may be used.

[0022] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 18, and is provided with a display panel inside that faces the eyes when worn. Light-emitting markers 110a, 110b, 110c, 110d, etc. are provided on the outer surface of the housing 108. The number and arrangement of the light-emitting markers are not particularly limited, but in this embodiment, they are provided at the four corners of the front of the housing of the output mechanism unit 102.

[0023] Furthermore, light-emitting markers 110e and 110f are also provided on both sides of the rear of the attachment band 106. By arranging the light-emitting markers in this manner, even if the user faces sideways or backwards with respect to the imaging device 12, the situation can be identified based on the number and positional relationship of the images of the light-emitting markers in the captured image. Note that light-emitting markers 110c and 110d are located below the output mechanism unit 102, and light-emitting markers 110e and 110f are located outside the attachment band 106. Since they are not actually visible from the viewpoint in Figure 2, their outer peripheries are indicated by dotted lines.

[0024] 3 shows the internal circuit configuration of the information processing device 10. The information processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are interconnected via a bus 30. An input / output interface 28 is also connected to the bus 30. Connected to the input / output interface 28 are a communication unit 32 including a peripheral device interface such as USB or IEEE1394 or a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or nonvolatile memory, an output unit 36 ​​that outputs data to the flat panel display 16 and HMD 18, an input unit 38 that inputs data from the imaging device 12, the input device 14, and the HMD 18, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, optical disk, or semiconductor memory.

[0025] The CPU 22 executes an operating system stored in the storage unit 34 to control the entire information processing device 10. The CPU 22 also executes various programs that have been read from a removable recording medium and loaded into the main memory 26, or that have been downloaded via the communication unit 32. The GPU 24 has the functions of a geometry engine and a rendering processor, performs drawing processing in accordance with drawing commands from the CPU 22, and stores display images in a frame buffer (not shown). The GPU 24 then converts the display images stored in the frame buffer into video signals and outputs them to the output unit 36. The main memory 26 is composed of RAM (Random Access Memory), and stores programs and data required for processing.

[0026] 4 shows the internal circuit configuration of the HMD 18. The HMD 18 includes a CPU 50, a main memory 52, a display unit 54, and an audio output unit 56. These units are connected to one another via a bus 58. An input / output interface 60 is also connected to the bus 58. A communication unit 62, which is formed by a wired or wireless LAN network interface, an acceleration sensor 64, and a light-emitting unit 66 are connected to the input / output interface 60.

[0027] The CPU 50 processes information acquired from each unit of the HMD 18 via the bus 58 and supplies output data to the display unit 54 and audio output unit 56. The main memory 52 stores programs and data necessary for processing by the CPU 50. However, depending on the application to be executed and the design of the device, it may be sufficient for the information processing device 10 to perform almost all processing, and for the HMD 18 to simply output data transmitted from the information processing device 10. In this case, the CPU 50 and main memory 52 can be replaced with simpler devices.

[0028] The display unit 54 is configured with a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the HMD 18. As described above, stereoscopic vision may be achieved by displaying a pair of parallax images in areas corresponding to the left and right eyes. The display unit 54 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the HMD 18 is worn, and that expand the user's field of view.

[0029] The audio output unit 56 is composed of speakers or earphones provided at positions corresponding to the user's ears when the HMD 18 is worn, and allows the user to hear audio. There is no particular limitation on the number of channels of the output audio, and it may be monaural, stereo, or surround. The communication unit 62 is an interface for sending and receiving data between the information processing device 10 and the flat display 16, and can be realized using a known wireless communication technology such as Bluetooth (registered trademark).

[0030] The acceleration sensor 64 detects the tilt of the HMD 18 by measuring the gravitational acceleration in a predetermined axis direction. The HMD 18 may also be provided with various other sensors, such as a gyro sensor. Measurement values ​​from the sensors are transmitted to the information processing device 10 via the communication unit 62. The light-emitting units 66 are elements or groups of elements that emit light in a predetermined color, and are provided at multiple locations on the outer surface of the HMD 18 as shown in FIG. 2. The position of the HMD 18 is obtained by tracking these as markers, and the orientation of the HMD 18 is also obtained from the number and positional relationship of their images in the captured image.

[0031] The information processing device 10 can acquire the position and posture of the user's head with higher accuracy by integrating information obtained by multiple means, such as the acceleration sensor 64 and the light-emitting unit 66. Meanwhile, in this embodiment, the acceleration sensor 64 can be omitted in some cases.

[0032] Fig. 5 shows the configuration of functional blocks in information processing device 10. Each functional block shown in Fig. 5 can be realized in terms of hardware by the configuration of the CPU, GPU, various memories, data bus, etc. shown in Fig. 3, and in terms of software by a program that performs various functions such as data input function, data storage function, image processing function, and communication function, loaded into memory from a recording medium, etc. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0033] The information processing device 10 includes an input information acquisition unit 72 that acquires input information from the input device 14 and the HMD 18, a captured image acquisition unit 74 that acquires captured image data from the imaging device 12, an information processing unit 76 that performs information processing according to content such as a game, an output data generation unit 78 that generates data to be output, and a content data storage unit 84 that stores data necessary for information processing and image generation. The information processing device 10 further includes a position and orientation information acquisition unit 80 that acquires information about the user's position and orientation based on the captured image or the like, a user guide generation unit 82 that generates information related to the user's situation in real space as a user guide, a guide data storage unit 85 that stores user guide presentation rules and data necessary for output, and an output data transmission unit 86 that transmits the data to be output to the HMD 18.

[0034] The input information acquisition unit 72 acquires the content of user operations from the input device 14. Here, user operations include operations performed on a general information processing device, such as selecting an application or content to be executed, starting / ending processing, and inputting commands, as well as operations on a user guide, such as displaying / hiding the user guide (described later) and switching the orientation of an image displayed as the user guide. The input information acquisition unit 72 supplies the information acquired from the input device 14 to the captured image acquisition unit 74, the information processing unit 76, or the user guide generation unit 82, depending on the content of the information. The input information acquisition unit 72 further receives measurement values ​​from the acceleration sensor 64 of the HMD 18 at a predetermined rate and supplies them to the position / orientation information acquisition unit 80.

[0035] The captured image acquisition unit 74 acquires, at a predetermined rate, data of captured images, such as stereo images, obtained by capturing a moving image by the imaging device 12. The captured image acquisition unit 74 may further control the start / end of imaging in the imaging device 12 in accordance with a processing start / end request from the user acquired by the input information acquisition unit 72, or may control the type of data acquired from the imaging device 12 in accordance with the result of processing in the information processing unit 76.

[0036] The position and orientation information acquisition unit 80 acquires information on the user's position and orientation at a predetermined rate by detecting images of predetermined objects from captured images. For example, the position of the user's head or hands in real space may be acquired based on images of luminous markers provided on the HMD 18 or the input device 14. Alternatively, image analysis techniques may be appropriately combined, such as tracking parts of the user's body using contours or detecting faces or objects with specific patterns using pattern matching. Depending on the configuration of the imaging device 12, the distance to the user may be determined by measuring infrared reflection, as described above. The position and orientation information acquisition unit 80 may further integrate measurement results from the acceleration sensor of the HMD 18 to more precisely identify the orientation of the user's head. Furthermore, as described below, depending on the content of the presented user guide, the position and orientation information acquisition unit 80 may also detect the position of subjects other than the user.

[0037] The information processing unit 76 processes electronic content, such as a game, designated by the user. This processing includes the use of information about the user's position and posture acquired by the position and posture information acquisition unit 80. As described above, the content of the subsequent information processing performed by the information processing unit 76 in response to user operations and movements via the input device 14 is not particularly limited.

[0038] The output data generation unit 78 generates image and audio data to be output as a result of information processing in accordance with a request from the information processing unit 76. For example, it generates left and right parallax images of the virtual world as seen from a viewpoint corresponding to the position and posture of the user's head. By displaying these parallax images in front of the left and right eyes on the HMD 18 and outputting audio in the virtual world, the user can feel as if they are immersed in the virtual world. The programs and image / audio data required for information processing in the information processing unit 76 and data generation processing in the output data generation unit 78 are stored in the content data storage unit 84.

[0039] The user guide generation unit 82 performs processing related to presenting a user guide to help the user confirm the surrounding situation in the real world and give suggestions on what actions to take afterwards, based on the information on the positions and postures of the user and other subjects acquired by the position and posture information acquisition unit 80. For example, information such as the relative position with respect to the imaging device 12, the preferred range of the user's position (hereinafter referred to as the "play area"), and the direction of movement to return to the center of the play area, etc., is presented in an easy-to-understand format, such as an illustration.

[0040] By superimposing a user guide showing such information on the game screen being played, the user can grasp the situation in the real world without removing the HMD, even in the middle of a game. The user guide generation unit 82 displays the user guide as needed in response to a user's operation to display the user guide. Alternatively, depending on the situation, such as to warn of danger, the user guide may be displayed regardless of the user's operation.

[0041] The angle of view of the camera of the imaging device 12, the corresponding play area settings, setting information such as conditions for outputting the user guide and the content to be presented, and image data required to generate the user guide are stored in the guide data storage unit 85 and read out as needed. Information other than the angle of view of the camera may be set in association with the content of the information processing performed by the information processing unit 76. The user guide generation unit 82 may present the user guide by audio in addition to the image display. In this case, audio data to be output depending on the situation is also stored in the guide data storage unit 85. Specific examples of user guides will be described later.

[0042] The user guide generation unit 82 supplies the generated user guide data to the output data generation unit 78. The user guide is designed to constantly reflect changes in the user's position and other circumstances. Therefore, the user guide generation unit 82 continues to generate and supply data at a predetermined rate while the conditions for displaying the user guide are met. The output data generation unit 78 superimposes the supplied user guide images and audio on images and audio generated as a result of information processing by the information processing unit 76. The output data transmission unit 86 sequentially acquires the output data generated by the output data generation unit 78, formats it as necessary, and transmits it to the HMD 18.

[0043] FIG. 6 is a diagram illustrating information that can be acquired from a captured image in this embodiment. In the figure, a user 120 holds an input device 14 and wears an HMD 18. The input device 14 has a light-emitting marker 122 at a position that directly faces the image capture device 12 when the input device 14 is held in a manner suitable for operation. The light-emitting markers of the HMD 18 are as shown in FIG. 2. If the image capture device 12 is a stereo camera, the distance Z from the imaging plane of the image capture device 12 to each light-emitting marker can be calculated based on the parallax of the images in the stereo image. Furthermore, the position of the image of the light-emitting marker in the image plane (XY plane) of either captured image represents the apparent position from the image capture device 12.

[0044] The position of each light-emitting marker in the three-dimensional space of the real world can be determined by integrating this information, specifically by back-projecting the position on the XY plane using the distance Z from the imaging device 12. Furthermore, the orientation (vector va) of the HMD 18 in real space can be determined from the number and positional relationship of the images of the light-emitting markers of the HMD 18, and the orientation (vector vb) of the input device 14 in real space can be determined from the shape of the images of the light-emitting markers 122 of the input device 14.

[0045] Based on the positions and orientations of these devices in real space, the information processing unit 76 of the information processing device 10 can express a virtual world in which the field of view changes depending on the direction of the user's 120's face, or the movement of an object in an image in response to the movement of the input device 14. Note that when estimating the distance from the image capture device 12 based on the apparent size of a marker, the image capture device 12 does not need to be a stereo camera. The same applies when distance measurement technology using reference light is introduced. Also, it is not necessary to track both the HMD 18 and the input device 14.

[0046] In this system, the user guide generator 82 recognizes the position and orientation of the HMD 18 as the user's position and head orientation, and illustrates the surrounding situation so that it can be understood from the user's perspective. Figure 7 shows an example of an image when a bird's-eye view is displayed as a user guide. Both of the user guides in Figures 7(a) and 7(b) are schematic diagrams of a bird's-eye view of the real space including the image capture device and the user, with Figure 7(a) being the image based on the position of the image capture device and Figure 7(b) being the image based on the orientation of the user. In both figures, a rectangle 130 representing the image capture device and a circle 132 representing the user are depicted to reflect the actual positional relationship and orientation.

[0047] A circle 132 representing the user is accompanied by a bar indicating the direction the user is actually facing. Both images also show an area 134 representing the horizontal play area determined by the camera's angle of view, etc., and an arrow 136 indicating the direction the user should move. The play area is basically set to match the camera's angle of view in the horizontal and vertical directions, or a narrower range of positions suitable for information processing. In addition, in the depth direction from the camera, a range is set that maintains sufficient distance acquisition accuracy, or a narrower range of positions suitable for information processing. The shape and size of the play area may be switched or selected by the user depending on the content of the information processing, such as the game being played, the size of the room, etc.

[0048] In the image (a), the rectangle 130 representing the imaging device is always centered at the top. This image allows the user to confirm that they are at the left edge of the play area, relatively far back from the imaging device 12. Moving any further to the left or back will result in straying from the play area, and the user can recognize the direction to move in to return to the center of the play area. As the user moves while this user guide is displayed, the circle 132 representing the user also moves in tandem. By displaying an overhead view that reflects the user's movements in this way, the relative positions of the imaging device, play area, and the user are clarified, and the stride length, direction, and other information, which may be subjective and ambiguous, are visually and objectively expressed. This allows the "scene" recognized by the information processing device 10 to be aligned with the user's own sense of movement.

[0049] In the case of an image such as (a), when the user is not facing the imaging device 12, the front, back, left, and right directions of the user themselves do not match the top, bottom, left, and right directions of the image. Therefore, even if the user tries to move in the direction of the arrow 136, it is conceivable that the user will not be able to do so. In contrast, the image in (b) is expressed so that the direction the user is facing is always vertically upward in the image. In this case, when the user's orientation changes, the field itself including the rectangle 130 and the area 134 will rotate in the opposite direction relative to the image. In this way, the front, back, left, and right directions of the user themselves always match the top, bottom, left, and right directions of the image, so the direction of the arrow 136 matches the user's sense of direction, making it easier to move accurately in that direction.

[0050] It should be noted that the figures of the imaging device and the like shown in the illustrated image are not limited to these. Furthermore, setting information such as the shape of the set play area, whether to guide using arrows or the like, and the direction of guidance if any, may vary depending on the content and situation of information processing by the information processing unit 76. Furthermore, depending on the content of guidance, the arrow may be a curve such as a U-shape. The length of the arrow may represent the distance to be traveled. Whether the reference for the orientation of the overhead view is based on the orientation of the imaging device or the user may be switched depending on the content of information processing, or may be switched by the user via the input device 14 or the like. The same applies to the user guides exemplified hereinafter.

[0051] Fig. 8 shows an example of a display screen on which a user guide image is displayed. The display screen 200 has a configuration in which a user guide 202, as shown in Fig. 7, is superimposed on an image of content such as a game screen. The user guide generation unit 82 displays the user guide 202 when the user performs an operation to display the user guide, and hides the user guide 202 when the user performs an operation to end the display. Alternatively, depending on the situation, the user guide may be displayed regardless of a user operation.

[0052] In either case, by superimposing the user guide on the original content image, the user can check the displayed information without moving their eyes significantly. On the other hand, if the content image is a virtual world with a sense of depth, the user may be surprised or feel uncomfortable when a component image with no sense of depth appears in front of them, especially if the content image is a virtual world with a sense of depth.

[0053] Therefore, as shown in the figure, the edges of the user guide 202 are blurred using blur processing or alpha blending, etc., to enhance the affinity with the virtual world depicted behind it, even though it is in the foreground. The entire user guide 202 may be made semi-transparent. Furthermore, unless the information displayed is urgent, it is desirable to display the user guide 202 in a position that avoids the point of gaze. For example, in a display format that changes the field of view according to the user's line of sight, the viewpoint is inevitably fixed at the center of the screen, so the user guide is displayed at a position at least a predetermined distance from the center of the screen.

[0054] Furthermore, since it is easier for people to lower their gaze than to raise it, by displaying the user guide 202 in the lower half of the screen (the area below the vertical center line C), it is possible to move the point of gaze from the content image without difficulty. Alternatively, by keeping the point of gaze fixed on the content image and checking the user guide 202 with the corner of your eye, it is possible to continue operating a game, etc. These measures can prevent the eyes from not immediately focusing on the displayed user guide, or to prevent motion sickness caused by small eye movements.

[0055] Figure 9 shows an example of a side view image used as a user guide. This example shows a simplified diagram of the real space, including the image capture device and the user, as seen from the user's left side. A rectangle 140 representing the image capture device and an object 142 representing the user are displayed to reflect actual distances and orientations. The image also shows a region 144 representing the vertical play area, determined by factors such as the camera's angle of view, and an arrow 146 indicating the direction of movement the user should take.

[0056] This type of display also makes it possible to grasp the situation as described with reference to FIG. 7. For example, the user can grasp that they are too close to the imaging device and their head is about to go out of the field of view, or that they are too far away and processing accuracy cannot be maintained. Furthermore, if posture is prescribed by the content of information processing, a user who is in a different posture can be urged to adopt the correct posture. For example, in a game that requires the user to be seated, if the user suddenly stands up, the arrow 146 can be displayed to urge the user to sit down.

[0057] Compared to the image shown in Figure 7, this example shows the user's figure as a more detailed object. This allows the user's actual posture to be reflected in detail, but depending on the content to be displayed, a simple figure like the example shown in Figure 7 may also be used. Also, animations that change such objects or figures may be used to encourage correct posture. For example, an animation of an expanding figure may encourage the user to stand up, or an animation of a shrinking figure may encourage the user to sit down.

[0058] Fig. 10 shows another example of an image in which a bird's-eye view is displayed as a user guide. In this example, similar to the image in Fig. 7, a rectangle 150 representing the imaging device and a circle 152 representing the user are displayed to reflect their actual positional relationship and orientation. However, in this figure, information related to the camera's angle of view is represented by lines 154 instead of the play area. Specifically, radial lines representing the orientations that divide the camera's angle of view into four equal parts and concentric arcs representing the distance from the imaging device at predetermined intervals are shown.

[0059] If the entire area within the camera's field of view is the play area, line 154 also indicates the approximate play area. For example, before executing information processing such as a game, such a user guide can be displayed while wearing HMD 18, and the user can actually move while looking at it, thereby understanding the correspondence between the amount and direction of their own movement relative to the camera's field of view and their sense of stride length, directional awareness, etc. By superimposing such a user guide on the virtual world displayed on the HMD, not just before information processing, the user can associate the scenery in the virtual world with their own position in the camera's field of view and act with that in mind.

[0060] While the illustrated example shows a bird's-eye view with the imaging device positioned above, a similar bird's-eye view may be used with the user facing vertically upward, as shown in FIG. 7(b). Furthermore, in a side view such as that shown in FIG. 9, a line equally dividing the vertical angle of view and a line representing the distance from the imaging device may be shown. Furthermore, a three-dimensional structure consisting of planes equally dividing the angle of view in the horizontal and vertical directions and planes equidistant from the imaging device may be defined, and the structure may be depicted as viewed from a desired viewpoint. The user's view of such a structure may be superimposed on the virtual world image displayed on the HMD 18, allowing the correspondence between the angle of view and the virtual world to be readily apparent.

[0061] FIG. 11 shows another example of an image in which a bird's-eye view is displayed as a user guide. In this example, the user's facing direction is always depicted as pointing vertically upward. Therefore, a circle 162 representing the user is displayed at the center of the bottom edge of the image, and a rectangle 160 representing the imaging device is displayed at a position that reflects its relative position to the user. Furthermore, in this figure, information relating to the user's field of view is represented by a line 164, instead of the information relating to the camera's angle of view in FIG. 10.

[0062] Specifically, the diagram shows radial lines representing the directions that divide the user's effective field of view into four equal parts, and concentric arcs representing the distance from the user at predetermined intervals. The effective field of view is a general numerical value that represents the angle at which the brain can accurately recognize an object. This user guide allows the user to easily understand what is in the user's field of view and where the imaging device is located in that field of view, even when the real world is not visible through the HMD. This allows the user to intuitively determine, for example, which direction and how much movement is required to face the imaging device. While the illustrated example shows a bird's-eye view based on the user's orientation, a bird's-eye view with the imaging device fixed on top, as in Figure 10, may also be used. The images shown in Figures 10 and 11 may also display arrows to guide the user.

[0063] FIG. 12 illustrates an example of an image in which a point cloud is displayed as a user guide. While the examples shown in FIGS. 7, 9, 10, and 11 are planar views of the real world viewed from above or from the side, the real world may be viewed obliquely to make it easier to grasp as a three-dimensional space. FIG. 12 shows an example in which a point cloud 172 representing a subject is displayed in a three-dimensional space corresponding to the real world, along with a figure 170 representing an imaging device, and is then rendered as an image. The point cloud 172 shown here is a collection of points representing the surface of a subject, obtained by inversely projecting a depth map of the subject, including the user, into three-dimensional space.

[0064] As described above, the depth map is an image in which the distance in the depth direction is acquired for each pixel based on the parallax of the stereo images and the acquired values ​​are expressed as pixel values, and is generated by the position and orientation information acquisition unit 80. The user guide generation unit 82 plots each pixel of the depth map in a virtual three-dimensional space based on the distance represented by the pixel value, thereby arranging the points that make up the point cloud 172 in a global coordinate system. Objects of the imaging device and a plane representing the floor surface are also arranged in the three-dimensional space. Then, by projecting these onto a predetermined screen coordinate system, an image as shown in the figure can be generated.

[0065] The depth map changes according to changes in the position and posture of the user or other subjects, and thus the point cloud 172 also changes. As a result, as in the examples described above, it is possible to objectively grasp one's own position and movement relative to the imaging device. Furthermore, by making it possible to accept user operations that change the position and posture of the screen surface, the user can confirm the relative position from an angle that is easy for the user to understand.

[0066] Furthermore, when creating a depth map by extracting an image from the entire stereo image, subjects other than the user are also drawn at the same time, so it is possible to check whether there are any obstacles in the surrounding area, or whether children or pets have entered the play area. Note that the example shown in the figure is composed only of a figure 170 representing the imaging device and a point cloud 172, but as in the previous examples, it is also possible to represent the play area or the camera's angle of view, or to use an arrow to indicate the direction in which the user should move.

[0067] FIG. 13 shows another example of an image in which an overhead view is displayed as a user guide. Similar to the image in FIG. 7, this example shows a rectangle 180 representing an imaging device, a circle 182 representing a user, and an area 184 representing a play area, all of which are displayed to reflect their actual positions and orientations. Furthermore, in this figure, the presence of objects other than the user within the play area is indicated by corresponding shapes. Specifically, a circle 186 representing an input device and a cross 188 representing an obstacle are displayed.

[0068] The positions of input devices and obstacles can be determined by extracting images other than the user's image using the depth map or background subtraction method described above. Input devices can be distinguished from other objects by their external luminous markers or the shape of the input device itself. Moving objects other than the user can be extracted by calculating the difference in movement from the previous frame. This user guide allows the user to navigate to and grasp the input device without removing the HMD, even if the user forgets the input device or needs it during information processing. Furthermore, even if there are obstacles nearby or if children or pets enter the play area, the device can recognize them and move them away, thereby avoiding collisions and other hazards.

[0069] In the example shown in the figure, a graphic representing an input device and a graphic representing an obstacle are shown simultaneously, but of course, either one may be shown depending on the situation. In particular, in dangerous situations, such as when an obstacle is detected in the play area, a user guide showing the location of the obstacle as a graphic may be displayed in a location that is easy for the user to notice, such as the center of the screen, regardless of user operation. It is desirable that the graphics representing the user, input device, and obstacle be in a form that can be distinguished at a glance. In addition to making the shapes significantly different as shown in the figure, they may also be displayed in different colors or with various other different processing methods.

[0070] Depending on the situation, the image may be displayed as an animation, such as blinking or slight vibration. For example, such animation can be used to alert the user when the user is about to leave the play area or when an obstacle enters the play area. If the image capture device 12 itself falls or changes direction due to some external force, the rectangle 180 representing the image capture device may be blinked to notify the user. Such changes in the image capture device 12 can be detected by the position and orientation information acquisition unit 80 or the user guide generation unit 82 by the captured image acquisition unit 74 separately acquiring measured values ​​from an acceleration sensor built into the image capture device 12.

[0071] Next, the operation of the information processing device 10, which can be realized by the configuration described above, will be described. Fig. 14 is a flowchart showing the processing procedure by which the information processing device generates output data according to the user's movements in this embodiment. This flowchart is started when the user requests the information processing device 10 to start processing via the input device 14, for example.

[0072] First, the captured image acquisition unit 74 of the information processing device 10 requests the imaging device 12 to start capturing images, and in response starts acquiring data of images captured and output by the imaging device 12 (S10). Meanwhile, the user guide generation unit 82 reads setting information such as the angle of view of the camera of the imaging device 12, the corresponding play area, the timing at which to display the user guide, and the content to be displayed accordingly, from the guide data storage unit 85 (S12). At this time, setting information associated with the content to be implemented, such as that selected by the user, may be selected and read.

[0073] Next, the position and orientation information acquisition unit 80 acquires information about the user's position and orientation based on the captured image (S14). The information processing unit 76 then performs information processing using this information, and the output data generation unit 78 draws a display image of the content as a result (S16). Meanwhile, the user guide generation unit 82 checks whether the current situation satisfies the conditions for displaying the user guide (S18). Conditions for displaying the user guide include when a user performs an operation to display the user guide, when there is a danger such as an obstacle in the play area, or when a factor that hinders the normal progress of information processing, such as an abnormality in the orientation of the imaging device 12, is detected.

[0074] The user guide generation unit 82, for example, sets a flag stored in an internal register or the like when such a state occurs, and lowers the flag when such a state is resolved, thereby branching the process based on the value of the flag. If the conditions for displaying a user guide are not met (N in S18), the output data transmission unit 86 outputs the image of the content drawn in S16 to the HMD 18 as is (S20). If the conditions for displaying a user guide are met (Y in S18), the user guide generation unit 82 generates an image of the user guide based on the position and orientation information acquired by the position and orientation information acquisition unit 80.

[0075] The data is then supplied to the output data generation unit 78, which then superimposes an image of the user guide on the image of the content (S20). The output data transmission unit 86 then outputs the image with the superimposed user guide to the HMD 18 (S22). Note that, in the process of S20, audio data of the user guide may also be superimposed on the audio data of the content. For example, specific verbal instructions such as "Please crouch" or "Please move to the right" may be used, or sound image localization technology may be used, such as generating a predetermined warning sound localized on the boundary line of the play area when the user approaches the boundary line.

[0076] The data output as the user guide may be a combination of images and audio, or just one of them. Depending on the settings, the images captured by the imaging device 12 may be output directly to the HMD 18 as the user guide. This allows the user to determine from the actual image whether there are any obstacles in the camera's field of view or whether there are any abnormalities in the camera's installation. In this case, the user guide generation unit 82 may superimpose text information such as "Is there anything dangerous?" and a GUI (Graphical User Interface) for the user to input their confirmation on the captured image, prompting the user to enter a confirmation.

[0077] When there is no need to terminate the process, such as when the user requests to stop the process, the content image is drawn based on the position information, and when the conditions are met, the user guide is superimposed and output to the HMD 18. This process is repeated at a predetermined rate (N in S24, S14 to S22). When it becomes necessary to terminate the process, all processes are terminated (Y in S24).

[0078] According to the embodiment described above, in a system that acquires user position information based on an image in a captured image and uses the acquired position information to process information, information related to the user's surroundings in the real world is presented as a user guide. This allows the user to recognize their current position and direction of movement in the real world even when wearing an HMD and unable to see the outside world. This reduces the possibility of moving outside the camera's field of view or the expected play area, causing problems with information processing, or colliding with objects outside the field of view. Furthermore, by simultaneously presenting the status of objects other than the user as a user guide, collisions within the play area can also be avoided.

[0079] Since the above confirmations can be made without removing the HMD, the burden on the user in recognizing the current situation and avoiding danger is reduced. Furthermore, by superimposing the information in a suitable position on the screen when necessary, it can be confirmed while viewing the image of the content displayed on the HMD, without disrupting the user's worldview. Furthermore, things that previously had to be relied on sensation, such as stride length and direction of movement, are visualized in correspondence with the position of the imaging device in the real world and the extent of the play area, making it easier to learn appropriate movements. Furthermore, since it uses position and posture information that is originally used for information processing, it can be achieved with a low processing load.

[0080] In addition, by allowing users to switch the orientation standard when displaying a bird's-eye view and the viewpoint when displaying 3D space, users can intuitively determine the direction they should move in with a display that is easy for them to understand.Furthermore, by providing users with movement guidelines using arrows and audio, it is possible to accommodate users of all ages and levels of understanding.

[0081] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and treatment processes, and that such modifications are also within the scope of the present invention.

[0082] For example, in the embodiment, an image that allows the user to understand the positional relationship of the user to the imaging device is presented as a user guide. However, when the user's position is identified using means other than the imaging device that captures the user, such as a camera or motion sensor attached to an HMD, the positional relationship with objects other than the imaging device may be displayed. For example, if a play area is set according to certain rules, it may be sufficient to simply indicate the user's position in the play area. Alternatively, if play equipment or other users are installed in the real space and the positional relationship with these is to be reflected in information processing, the positions of these objects or people may be displayed in the user guide instead of the imaging device. [Explanation of symbols]

[0083] 8 Information processing system, 10 Information processing device, 12 Imaging device, 14 Input device, 16 Flat display, 18 HMD, 22 CPU, 24 GPU, 26 Main memory, 72 Input information acquisition unit, 74 Captured image acquisition unit, 76 Information processing unit, 78 Output data generation unit, 80 Position and orientation information acquisition unit, 82 User guide generation unit, 85 Guide data storage unit, 86 Output data transmission unit.

Claims

1. a position information acquisition unit that acquires position information of the head mounted display; an information processing unit that processes information using the position information; an output data generation unit that generates and outputs data of a display image of the virtual world to be displayed as a result of the information processing; a user guide generation unit that generates image data of a user guide that semi-transparently displays a three-dimensional space in the real world where the user is located; Equipped with The information processing device is characterized in that the output data generation unit superimposes an image of the user guide on a display image of the virtual world while a condition for displaying the user guide is satisfied.

2. 2 . The information processing apparatus according to claim 1 , wherein the output data generating unit superimposes an image of the user guide when a user operation for displaying the user guide is performed as the condition.

3. The information processing apparatus according to claim 1 , wherein the output data generating unit superimposes the image of the user guide when a result of image recognition using the image of the real world satisfies the condition.

4. 4. The information processing device according to claim 1, wherein the user guide generation unit distinguishes between a play area and other areas in the real world in different colors in the user guide.

5. 4. The information processing device according to claim 1, wherein the user guide generation unit represents a boundary of a play area in the real world in the user guide.

6. 4. The information processing apparatus according to claim 1, wherein the user guide generating unit displays, in the user guide, how a three-dimensional structure that divides a distance in the real world equally is viewed from a viewpoint.

7. 4. The information processing device according to claim 1, wherein the user guide generating unit displays lines in the user guide that divide a distance equally in the horizontal and / or vertical directions in the real world.

8. the position information acquisition unit generates a depth map in which a pixel value represents a distance from an imaging device that captures the real world to a subject; 8. The information processing apparatus according to claim 1, wherein the user guide generating unit acquires the position of a real object other than the user using the depth map and displays the position in the user guide.

9. The information processing device according to any one of claims 1 to 7, characterized in that the user guide generation unit obtains the positions of real objects other than the user based on images extracted from an image of the real world using a background subtraction method, and displays the positions in the user guide.

10. 10. The information processing apparatus according to claim 1, wherein the user guide generating unit indicates, in the user guide, a direction in which the user is actually facing.

11. 11. The information processing device according to claim 1, wherein the output data generating unit blurs the periphery of the user guide at a boundary between the display image of the virtual world and the image of the user guide.

12. 12. The information processing device according to claim 1, wherein the position information acquisition unit acquires the position information by an imaging device attached to the head-mounted display.

13. acquiring position information of a head mounted display; performing information processing using the location information; generating and outputting data of a display image of the virtual world to be displayed as a result of the information processing; generating data of an image of a user guide that semi-transparently displays a three-dimensional space of the real world in which the user is located; Including, A method for presenting a user guide by an information processing device, characterized in that the output step superimposes an image of the user guide on a display image of the virtual world for a period of time during which the conditions for displaying the user guide are met.

14. A function to acquire position information of the head-mounted display, a function of performing information processing using the location information; a function of generating and outputting data of a virtual world display image to be displayed as a result of the information processing; A function to generate image data for a user guide that shows a semi-transparent three-dimensional space in the real world where the user is located. This is realized by a computer, The computer program is characterized in that the output function superimposes an image of the user guide on a display image of the virtual world while the conditions for displaying the user guide are met.

15. A head-mounted display that displays an image generated by an information processing device, a communication unit for transmitting and receiving data to and from the information processing device; a display unit that displays a virtual world display image generated as a result of information processing performed by the information processing device using the position information of the head-mounted display, and that displays a user guide image, which semi-transparently represents a three-dimensional space of the real world in which the user is located, superimposed on the virtual world display image while a condition for displaying the user guide is satisfied; A head-mounted display comprising:

Images