Information processing apparatus, user guide presentation method, and head mounted display

Abstract

To solve the problem that it is difficult for a user wearing an HMD to recognize a state of a real world.SOLUTION: As a user guide, a state where a real space including an imaging apparatus and a user is overlooked is represented by an image (a) based on a position of the imaging apparatus or an image (b) based on an orientation of the user. In both of the images, a rectangle 130 indicating the imaging apparatus and a circle 132 indicating the user are represented so as to reflect actual physical relation and orientations. Further, an area 134 indicating a play area in a horizontal direction, which is determined by a field angle of a camera and so on and an arrow 136 indicating a direction in which the user should move are represented.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device that processes information based on a captured image, a user guide presentation method performed by the information processing device, and a head mounted display that displays an image. [Background technology]

[0002] A head mounted display (hereinafter referred to as "HMD") connected to a game machine is worn on the head, and a game is played while looking at the displayed screen (see, for example, Patent Document 1). For example, if the position and posture of the user's head are acquired, and an image of the virtual world is displayed so that the field of view changes according to the direction of the face, it is possible to create a situation in which the user feels as if they have entered the virtual world. The position and posture of the user are generally acquired based on the analysis results of visible light or infrared images captured of the user, or measurements from a motion sensor built into the HMD. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5580855 specification Summary of the Invention [Problem to be solved by the invention]

[0004] Technologies that perform some kind of information processing based on captured images assume that the user or other object is within the camera's field of view. Depending on the content of the information processing, the area in which the user should be is further limited. However, when wearing an HMD, the user cannot see the outside world, so they may lose their sense of direction or become so engrossed in the game that they move to an unexpected position in the real world. This can cause problems such as a deterioration or breakdown in the accuracy of information processing, or collisions with other objects, but the cause is difficult to find unless the HMD is removed. If such situations occur frequently, the worldview of the virtual space constructed using the HMD will be hindered.

[0005] The present invention has been made in consideration of these problems, and its purpose is to provide technology that allows users to continue to enjoy the world displayed by an HMD with minimal burden. [Means for solving the problem]

[0006] One aspect of the present invention relates to an information processing device, which includes a position information acquisition unit that acquires position information of a head mounted display, an information processing unit that processes information using the position information, an output data generation unit that generates and outputs data of a display image of a virtual world to be displayed as a result of the information processing, and a user guide generation unit that generates data of an image of a user guide that semi-transparently represents the state of the real world in which the user is located, and is characterized in that the output data generation unit superimposes the image of the user guide on the display image of the virtual world while a condition for displaying the user guide is satisfied.

[0007] Another aspect of the present invention relates to a user guide presentation method, the user guide presentation method including a step of acquiring position information of a head mounted display, a step of performing information processing using the position information, a step of generating and outputting data of a display image of a virtual world to be displayed as a result of the information processing, and a step of generating data of an image of a user guide that semi-transparently represents a state of the real world in which a user exists, the outputting step being characterized in that the image of the user guide is superimposed on the display image of the virtual world during a period in which a condition for displaying the user guide is satisfied.

[0008] Yet another aspect of the present invention relates to a head mounted display, which displays an image generated by an information processing device, and is characterized by including: a communication unit for transmitting and receiving data to and from the information processing device; and a display unit for displaying a display image of a virtual world generated as a result of information processing performed by the information processing device using position information of the head mounted display, and for displaying an image of a user guide, which semi-transparently represents the state of the real world in which the user is located, superimposed on the display image of the virtual world while a condition for displaying the user guide is satisfied.

[0009] Any combination of the above components, and any conversion between the expression of the present invention, such as a method, an apparatus, a system, a computer program, or a recording medium having a computer program recorded thereon, are also effective as aspects of the present invention. Effect of the Invention

[0010] According to the present invention, a user wearing an HMD can continue to enjoy the world of expression with minimal burden. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing system to which the present embodiment can be applied. [Diagram 2] 2A to 2C are diagrams illustrating examples of the external shape of an HMD according to the present embodiment. [Diagram 3] 1 is a diagram showing an internal circuit configuration of an information processing device according to an embodiment of the present invention; [Figure 4] 2 is a diagram showing the internal circuit configuration of an HMD according to the present embodiment. [Diagram 5] 1 is a diagram showing a functional block configuration of an information processing device according to an embodiment of the present invention; [Figure 6] 1 is a diagram for explaining information that can be acquired from a captured image in the present embodiment. FIG. [Figure 7] 11 is a diagram illustrating an example of an image when an overhead view is displayed as a user guide in the present embodiment. FIG. [Figure 8] 11 is a diagram illustrating an example of a display screen on which an image of a user guide is displayed in the present embodiment. FIG. [Figure 9] 11A and 11B are diagrams illustrating examples of images when a side view is displayed as a user guide in the present embodiment. [Figure 10] 13 is a diagram showing another example image when an overhead view is displayed as a user guide in the present embodiment. FIG. [Figure 11] 13 is a diagram showing another example image when an overhead view is displayed as a user guide in the present embodiment. FIG. [Figure 12] 11 is a diagram illustrating an example of an image in which a point cloud is displayed as a user guide in this embodiment. FIG. [Figure 13] 13 is a diagram showing another example image when an overhead view is displayed as a user guide in the present embodiment. FIG. [Figure 14] 10 is a flowchart showing a processing procedure in which the information processing device generates output data according to a user's movement in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1 shows an example of the configuration of an information processing system to which this embodiment can be applied. The information processing system 8 includes an imaging device 12 that captures an image of an object, an information processing device 10 that processes information based on the captured image, a flat panel display 16 and an HMD 18 that display an image obtained as a result of the information processing, and an input device 14 that is operated by a user.

[0013] The information processing device 10, the imaging device 12, the input device 14, the flat display 16, and the HMD 18 may be connected by wired cables or by known wireless communication technology such as Bluetooth (registered trademark). Depending on the information processing performed by the information processing device 10, the flat display 16 may not be required. The external shapes of these devices are not limited to those shown in the drawings. Furthermore, a device may be provided that is equipped with two or more of these devices in an integrated manner. For example, the information processing device 10, the input device 14, and the flat display 16 may be realized by a mobile terminal equipped with them.

[0014] The imaging device 12 has a camera that captures an object such as a user at a predetermined frame rate, and a mechanism that generates output data of the captured image by performing general processing such as demosaic processing on the output signal from the camera and sends the output data to the information processing device 10. The camera has a visible light sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor that is used in general digital cameras and digital video cameras. The imaging device 12 may have only one camera, or may be a so-called stereo camera in which two cameras are arranged on the left and right at a known distance as shown in the figure.

[0015] Alternatively, the imaging device 12 may be configured by combining a monocular camera with a device that irradiates a reference light such as infrared light onto an object and measures the reflected light. When a stereo camera or a mechanism for measuring reflected light is introduced, the position of the object can be found in three-dimensional real space, and information processing by the information processing device 10 and image display by the display device can be further diversified. Both the method of determining the distance of the object from the camera based on the principle of triangulation using stereo images captured by a stereo camera from left and right viewpoints, and the method of determining the distance of the object from the camera based on the TOF (Time of Flight) or pattern irradiation method by measuring the reflection of the reference light are widely known.

[0016] Hereinafter, the imaging device 12 will be mainly described as capturing stereo images, but as described above, this is not intended to limit the present embodiment, and the imaging device 12 may include at least one camera. The information processing device 10 performs necessary information processing using data transmitted from the imaging device 12, and generates output data such as images and sounds. The content of the processing performed by the information processing device 10 here is not particularly limited, and may be appropriately determined depending on the functions, applications, electronic contents, etc. desired by the user.

[0017] For example, the information processing device 10 performs general face detection and tracking processing on the captured image to progress a game in which a character reflecting the user's movements as a target appears, or converts the user's movements into command inputs to perform information processing. At this time, markers provided on the input device 14 may be used to acquire the movements of the input device 14. In addition, by tracking multiple markers provided on the outer surface of the HMD 18, the position and posture of the head of the user wearing the HMD 18 may be identified, and the virtual world viewed from a viewpoint that moves in response to the position and posture may be displayed on the HMD 18. The output data generated by the information processing device 10 is transmitted to at least the HMD 18.

[0018] The HMD 18 is a display device that displays an image on a display panel such as an organic EL panel located in front of the user's eyes when worn on the user's head. For example, parallax images viewed from left and right viewpoints may be generated and displayed in the left and right areas obtained by dividing the display screen into two, thereby allowing the user to view the image stereoscopically. However, this is not intended to limit the present embodiment, and one image may be displayed on the entire display screen. The HMD 18 may further include built-in speakers or earphones that output sound to positions corresponding to the user's ears.

[0019] The flat display 16 may be a television having a display for outputting two-dimensional images and a speaker for outputting sound, such as a liquid crystal television, an organic electroluminescence television, a plasma television, a PC display, or the like. Alternatively, it may be the display and speaker of a tablet terminal or a mobile terminal. The input device 14 is operated by a user to receive requests such as starting and ending a process, selecting a function, and inputting various commands, and supplies the requests to the information processing device 10 as electrical signals.

[0020] The input device 14 may be realized by any of common input devices such as a game controller, a keyboard, a mouse, a joystick, a touch pad provided on the display screen of the flat display 16, or a combination of these. The input device 14 may further include a light-emitting marker consisting of an element that emits light in a predetermined color or a collection of such an element. In this case, the information processing device 10 tracks the movement of the marker using the captured image, so that the movement of the input device 14 itself can be treated as a user operation. The input device 14 may be configured only by a light-emitting marker and a mechanism for holding it.

[0021] 2 shows an example of the external shape of the HMD 18. In this example, the HMD 18 is composed of an output mechanism 102 and a wearing mechanism 104. The wearing mechanism 104 includes a wearing band 106 that goes around the head when worn by the user to secure the device. The wearing band 106 is made of a material or has a structure that allows the length to be adjusted to fit the head circumference of each user. For example, it may be made of an elastic body such as rubber, or a buckle, gears, etc. may be used.

[0022] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 18, and has a display panel inside so as to face the eyes when worn. Light-emitting markers 110a, 110b, 110c, 110d, etc. are provided on the outer surface of the housing 108. The number and arrangement of the light-emitting markers are not particularly limited, but in this embodiment, they are provided at the four corners of the front surface of the housing of the output mechanism unit 102.

[0023] Furthermore, light-emitting markers 110e and 110f are provided on both sides of the rear of the attachment band 106. By arranging the light-emitting markers in this manner, even if the user faces sideways or backwards with respect to the imaging device 12, the situation can be identified based on the number and positional relationship of the images of the light-emitting markers in the captured image. Note that the light-emitting markers 110c and 110d are located below the output mechanism unit 102, and the light-emitting markers 110e and 110f are located outside the attachment band 106. Since they are not actually visible from the viewpoint in FIG. 2, their outer peripheries are indicated by dotted lines.

[0024] 3 shows the internal circuit configuration of the information processing device 10. The information processing device 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 is connected to a communication unit 32 including a peripheral device interface such as USB or IEEE1394, or a network interface for wired or wireless LAN, a storage unit 34 such as a hard disk drive or nonvolatile memory, an output unit 36 ​​that outputs data to the flat panel display 16 or HMD 18, an input unit 38 that inputs data from the imaging device 12, the input device 14, and the HMD 18, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0025] The CPU 22 executes an operating system stored in the storage unit 34 to control the entire information processing device 10. The CPU 22 also executes various programs read from a removable recording medium and loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has a geometry engine function and a rendering processor function, performs drawing processing according to drawing commands from the CPU 22, and stores a display image in a frame buffer (not shown). The GPU 24 then converts the display image stored in the frame buffer into a video signal and outputs it to the output unit 36. The main memory 26 is composed of a RAM (Random Access Memory), and stores programs and data required for processing.

[0026] 4 shows the internal circuit configuration of the HMD 18. The HMD 18 includes a CPU 50, a main memory 52, a display unit 54, and an audio output unit 56. These units are connected to one another via a bus 58. An input / output interface 60 is further connected to the bus 58. A communication unit 62, which is formed of a wired or wireless LAN network interface, an acceleration sensor 64, and a light-emitting unit 66 are connected to the input / output interface 60.

[0027] The CPU 50 processes information acquired from each unit of the HMD 18 via the bus 58, and supplies output data to the display unit 54 and the audio output unit 56. The main memory 52 stores programs and data necessary for processing in the CPU 50. However, depending on the application to be executed and the design of the device, it may be sufficient for the information processing device 10 to perform almost all processing, and for the HMD 18 to only output data transmitted from the information processing device 10. In this case, the CPU 50 and the main memory 52 can be replaced with simpler devices.

[0028] The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the HMD 18. As described above, stereoscopic vision may be realized by displaying a pair of parallax images in areas corresponding to the left and right eyes. The display unit 54 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the HMD 18 is worn, and that expand the user's viewing angle.

[0029] The audio output unit 56 is composed of speakers or earphones provided at positions corresponding to the user's ears when the HMD 18 is worn, and allows the user to hear audio. The number of channels of the output audio is not particularly limited, and may be any of monaural, stereo, and surround. The communication unit 62 is an interface for transmitting and receiving data between the information processing device 10 and the flat display 16, and can be realized by using a known wireless communication technology such as Bluetooth (registered trademark).

[0030] The acceleration sensor 64 detects the inclination of the HMD 18 by measuring the gravitational acceleration in a predetermined axis direction. The HMD 18 may further be provided with various other sensors such as a gyro sensor. Measurement values ​​by the sensor are transmitted to the information processing device 10 via the communication unit 62. The light emitting units 66 are elements or groups of elements that emit light in a predetermined color, and are provided at multiple locations on the outer surface of the HMD 18 as shown in FIG. 2. The position of the HMD 18 is obtained by tracking these as markers, and the attitude of the HMD 18 is obtained from the number and positional relationship of their images in the captured image.

[0031] The information processing device 10 can obtain the position and posture of the user's head with higher accuracy by integrating information obtained by a plurality of means such as the acceleration sensor 64 and the light emitting unit 66. Meanwhile, in this embodiment, the acceleration sensor 64 can be omitted in some cases.

[0032] Fig. 5 shows the configuration of functional blocks of the information processing device 10. Each functional block shown in Fig. 5 can be realized in terms of hardware by the configuration of the CPU, GPU, various memories, data bus, etc. shown in Fig. 3, and in terms of software by a program that performs various functions such as a data input function, a data holding function, an image processing function, and a communication function, which are loaded into memory from a recording medium, etc. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0033] The information processing device 10 includes an input information acquisition unit 72 that acquires input information from the input device 14 and the HMD 18, a captured image acquisition unit 74 that acquires captured image data from the imaging device 12, an information processing unit 76 that performs information processing according to content such as a game, an output data generation unit 78 that generates data to be output, and a content data storage unit 84 that stores data necessary for information processing and image generation. The information processing device 10 further includes a position / posture information acquisition unit 80 that acquires information on the position and posture of a user based on the captured image or the like, a user guide generation unit 82 that generates information related to the user's situation in real space as a user guide, a guide data storage unit 85 that stores presentation rules for the user guide and data necessary for output, and an output data transmission unit 86 that transmits data to be output to the HMD 18.

[0034] The input information acquisition unit 72 acquires the contents of user operations from the input device 14. Here, the user operations include operations performed in a general information processing device, such as selection of an application or content to be executed, start / end of processing, and command input, as well as operations on a user guide, such as display / hide of a user guide, which will be described later, and switching of the orientation of an image displayed as the user guide. The input information acquisition unit 72 supplies the information acquired from the input device 14 to the captured image acquisition unit 74, the information processing unit 76, or the user guide generation unit 82 according to the contents of the information. The input information acquisition unit 72 further receives measurement values ​​from the acceleration sensor 64 of the HMD 18 at a predetermined rate and supplies them to the position / orientation information acquisition unit 80.

[0035] The captured image acquisition unit 74 acquires, at a predetermined rate, data of captured images, such as stereo images, obtained by the imaging device 12 capturing a moving image. The captured image acquisition unit 74 may further control the start / end of imaging in the imaging device 12 according to a processing start / end request from a user acquired by the input information acquisition unit 72, or may control the type of data acquired from the imaging device 12 according to the result of processing in the information processing unit 76.

[0036] The position and orientation information acquisition unit 80 acquires information on the position and orientation of the user at a predetermined rate by detecting the image of a predetermined object from the captured image. For example, the position of the user's head or hand in real space is acquired based on the image of a light-emitting marker provided on the HMD 18 or the input device 14. Alternatively, an image analysis technique may be appropriately combined to track a part of the user's body using a contour line or to detect a face or an object having a specific pattern by pattern matching. Depending on the configuration of the imaging device 12, the distance to the user may be specified by measuring the reflection of infrared rays as described above. The position and orientation information acquisition unit 80 may further integrate the measurement results by the acceleration sensor of the HMD 18 to specify the orientation of the user's head in more detail. Furthermore, as will be described later, depending on the content of the user guide to be presented, the position and orientation information acquisition unit 80 may detect the position of a subject other than the user.

[0037] The information processing unit 76 processes electronic content such as a game specified by the user. This processing includes the use of information on the user's position and posture acquired by the position and posture information acquisition unit 80. As described above, the content of the subsequent information processing performed by the information processing unit 76 in response to the user's operation and movement via the input device 14 is not particularly limited.

[0038] The output data generating unit 78 generates image and audio data to be output as a result of information processing in accordance with a request from the information processing unit 76. For example, the output data generating unit 78 generates a left and right parallax image of the virtual world as seen from a viewpoint corresponding to the position and posture of the user's head. If the parallax image is displayed in front of the left and right eyes on the HMD 18 and audio in the virtual world is output, the user can feel as if he or she has entered the virtual world. Programs and image / audio data required for information processing in the information processing unit 76 and data generation processing in the output data generating unit 78 are stored in the content data storage unit 84.

[0039] The user guide generating unit 82 performs processing related to presenting a user guide to allow the user to confirm the surrounding situation in the real world and to give suggestions on actions to be taken afterwards, based on the information on the positions and postures of the user and other subjects acquired by the position and posture information acquiring unit 80. For example, information such as the relative position with respect to the imaging device 12, the preferred range of the user's position (hereinafter referred to as the "play area"), and the direction of movement to return to the center of the play area, etc., is presented in an easy-to-understand format, such as in the form of an illustration.

[0040] By displaying a user guide showing such information superimposed on the game screen being played, the user can grasp the situation in the real world without removing the HMD, even in the middle of the game. The user guide generating unit 82 displays the user guide at any time in response to the user's operation to display the user guide. Alternatively, depending on the situation, such as warning of danger, the user guide may be displayed regardless of the user's operation.

[0041] The angle of view of the camera of the imaging device 12, the corresponding play area settings, setting information such as conditions for outputting a user guide and contents to be presented, image data required for generating a user guide, etc. are stored in the guide data storage unit 85 and read out as necessary. Information other than the angle of view of the camera may be set in association with the contents of information processing performed by the information processing unit 76. The user guide generating unit 82 may present the user guide by sound in addition to image display. In this case, sound data to be output according to the situation is also stored in the guide data storage unit 85. Specific examples of user guides will be described later.

[0042] The user guide generation unit 82 supplies the generated user guide data to the output data generation unit 78. The user guide is designed to always reflect changes in the situation, such as the user's position. Therefore, the user guide generation unit 82 continues to generate and supply data at a predetermined rate while the conditions for displaying the user guide are met. The output data generation unit 78 superimposes the supplied user guide images and sounds on images and sounds generated as a result of information processing by the information processing unit 76. The output data transmission unit 86 sequentially acquires the output data generated by the output data generation unit 78, shapes it as necessary, and transmits it to the HMD 18.

[0043] FIG. 6 is a diagram for explaining information that can be acquired from a captured image in this embodiment. In the figure, a user 120 holds an input device 14 and wears an HMD 18. The input device 14 is provided with a light-emitting marker 122 at a position directly facing the imaging device 12 when the input device 14 is held in a manner suitable for operation. The light-emitting markers of the HMD 18 are as shown in FIG. 2. When the imaging device 12 is a stereo camera, a distance Z from the imaging plane of the imaging device 12 to each light-emitting marker is obtained based on the parallax of the images in the stereo images. In addition, the position of the image of the light-emitting marker in the image plane (XY plane) of either captured image represents the apparent position from the imaging device 12.

[0044] The position of each light-emitting marker in the three-dimensional space of the real world is obtained by integrating this information, specifically by inversely projecting the position on the XY plane using the distance Z from the imaging device 12. Furthermore, the orientation (vector va) of the HMD 18 in real space can be obtained from the number and positional relationship of the images of the light-emitting markers of the HMD 18, and the orientation (vector vb) of the input device 14 in real space can be obtained from the shape of the images of the light-emitting markers 122 of the input device 14.

[0045] Based on the positions and orientations of these devices in real space, the information processing unit 76 of the information processing device 10 can express a virtual world in which the field of view changes according to the direction of the user's 120's face, and a state in which an object in an image moves according to the movement of the input device 14. When estimating the distance from the imaging device 12 based on the apparent size of a marker, the imaging device 12 does not need to be a stereo camera. The same applies when a distance measurement technique using a reference light is introduced. In addition, it is not necessary to track both the HMD 18 and the input device 14.

[0046] In such a system, the user guide generating unit 82 recognizes the position and orientation of the HMD 18 as the user's position and head orientation, and illustrates the surrounding situation so that it can be understood from the user's perspective. Figure 7 shows an example of an image when a bird's-eye view is displayed as a user guide. Both of the user guides in Figures 7(a) and 7(b) are schematic diagrams showing an overhead view of the real space including the imaging device and the user, with (a) being an image based on the position of the imaging device, and (b) being an image based on the orientation of the user. In both figures, a rectangle 130 representing the imaging device and a circle 132 representing the user are displayed so as to reflect the actual positional relationship and orientation.

[0047] A circle 132 representing the user is accompanied by a bar indicating the direction in which the user is actually facing. Both images also show an area 134 representing the horizontal play area determined by the camera's angle of view, etc., and an arrow 136 indicating the direction in which the user should move. The play area is basically set to match the camera's angle of view in the horizontal and vertical directions, or a narrower range of positions suitable for information processing is set. In addition, in the depth direction from the camera, a range that can maintain sufficient accuracy in obtaining distance, or a narrower range of positions suitable for information processing is set. The shape and size of the play area may be switched or selected by the user depending on the content of the information processing, such as a game to be played, the size of the room, etc.

[0048] The image (a) shows a rectangle 130 representing the imaging device always fixed at the top center. This image allows the user to confirm that he / she is at the left edge of the play area, relatively behind the imaging device 12. If he / she moves further to the left or back, he / she will deviate from the play area, and he / she can recognize the direction of movement to return to the center of the play area. When the user actually moves with such a user guide displayed, the circle 132 representing the user also moves in tandem. By displaying an overhead view reflecting the user's movement in this way, the relative positions of the imaging device, the play area, and the user become clear, and the stride length, direction, etc., which are sensory and ambiguous, are visually and objectively expressed. This allows the "scene" recognized by the information processing device 10 to match the user's own sense of movement.

[0049] In the case of the representation as shown in (a), when the user is not facing the imaging device 12, the front, back, left, right and top, bottom, left, and right of the user themselves do not match. Therefore, even if the user wants to move in the direction of the arrow 136, it is considered that the user cannot move in that direction. In contrast, the image in (b) is represented so that the direction in which the user is facing is always vertically upward in the image. In this case, when the user's orientation changes, the field itself including the rectangle 130 and the area 134 rotates relatively in the opposite direction. In this way, the front, back, left, right and top, bottom, left, and right of the user themselves always match, so that the direction of the arrow 136 matches the user's sense of direction, making it easier to move accurately in that direction.

[0050] The figures of the imaging device and the like shown in the illustrated image are not limited to these. Also, the setting information, such as the shape of the play area to be set, whether to guide with an arrow or the like, and the direction of the guidance if so, may vary depending on the contents and circumstances of the information processing by the information processing unit 76. Furthermore, depending on the contents of the guidance, the arrow may be a curve such as a U-shape. The length of the arrow may represent the distance to be traveled. Whether the reference for the orientation of the overhead view is the imaging device or the user's orientation may be switched depending on the contents of the information processing, or may be switched by the user via the input device 14 or the like. The same applies to the user guides exemplified hereinafter.

[0051] Fig. 8 shows an example of a display screen on which an image of a user guide is displayed. The display screen 200 has a configuration in which a user guide 202 as shown in Fig. 7 is superimposed on an image of a content such as a game screen. The user guide generating unit 82 displays the user guide 202 when the user performs an operation to display the user guide, and hides the user guide 202 when the user performs an operation to end the display. Alternatively, the user guide may be displayed regardless of the user's operation depending on the situation.

[0052] In any case, by superimposing the user guide on the original content image, the user can check the displayed information without moving his / her eyes significantly. On the other hand, if the content image is a virtual world with a sense of depth, the user may feel surprised or uncomfortable when a component image with no sense of depth appears in front of the user.

[0053] Therefore, as shown in the figure, the edges of the user guide 202 are blurred using blur processing or alpha blending, etc., to enhance affinity with the virtual world depicted behind it, even though it is in the foreground. The entire user guide 202 may be made semi-transparent. Furthermore, unless the information displayed is urgent, it is desirable to display the user guide 202 in a position that avoids the gaze point. For example, in a display format in which the field of view changes according to the user's line of sight, the viewpoint is inevitably fixed to the center of the screen, so the user guide is displayed at a position at least a predetermined distance away from the center of the screen.

[0054] Also, since it is easier for people to lower their gaze than to raise it, by displaying the user guide 202 in the lower half of the screen (the area below the vertical center line C), the gaze point can be moved effortlessly from the content image. Alternatively, it is possible to keep the gaze point on the content image and continue playing the game by checking the user guide 202 with the corner of the eye. These measures can prevent the eyes from not immediately focusing on the displayed user guide, or to prevent motion sickness caused by small eye movements.

[0055] 9 shows an example of an image in which a side view is displayed as a user guide. In this example, a schematic diagram of the real space including the imaging device and the user is shown as seen from the left side of the user. That is, a rectangle 140 representing the imaging device and an object 142 representing the user are shown to reflect the actual distance and orientation. The image also shows an area 144 representing the vertical play area, which is determined by the camera's angle of view, and an arrow 146 representing the direction of movement the user should take.

[0056] This display also makes it possible to grasp the situation as described with reference to FIG. 7. For example, the user can grasp that the user is too close to the imaging device and the head is about to go out of the field of view, or is too far away and processing accuracy cannot be maintained. Furthermore, when the posture is regulated by the contents of the information processing, the user who is in a different posture can be urged to take the correct posture. For example, in a game that requires the user to be seated, if the user suddenly stands up, the arrow 146 can be displayed to urge the user to sit down.

[0057] In this example, the user's figure is shown as a more detailed object compared to the image shown in Fig. 7. This allows the user's actual posture to be reflected in detail, but depending on the content to be displayed, a simple figure as shown in Fig. 7 may be used. Also, animations that change such objects or figures may be used to encourage the user to adopt a correct posture. For example, an animation in which a figure expands may be used to encourage the user to stand up, and an animation in which a figure shrinks may be used to encourage the user to sit down.

[0058] Fig. 10 shows another example of an image in which an overhead view is displayed as a user guide. In this example, as in the image of Fig. 7, a rectangle 150 representing an imaging device and a circle 152 representing a user are displayed so as to reflect the actual positional relationship and orientation. Meanwhile, in this figure, information related to the camera's angle of view is represented by lines 154 instead of the play area. Specifically, radial lines representing the orientations that divide the camera's angle of view into four equal parts and concentric arcs representing the distance from the imaging device at predetermined intervals are represented.

[0059] If the entire area within the camera's field of view is the play area, the line 154 also indicates the approximate play area. For example, before executing information processing such as a game, such a user guide is displayed while wearing the HMD 18, and the user actually moves while looking at it, allowing the user to understand the correspondence between the amount and direction of one's movement relative to the camera's field of view and the senses such as stride length and directional recognition. By displaying such a user guide superimposed on the virtual world displayed on the HMD, not only before information processing, the user can associate the scenery of the virtual world with one's position in the camera's field of view and act with that in mind.

[0060] Although the illustrated example is an overhead view with the imaging device placed on the top, it may be an overhead view with the user facing vertically upward as shown in FIG. 7(b) with a similar configuration. Also, in a side view such as FIG. 9, a line dividing the vertical angle of view equally and a line representing the distance from the imaging device may be shown. Furthermore, a three-dimensional structure consisting of a plane dividing the angle of view equally in the horizontal and vertical directions and a plane equidistant from the imaging device may be defined, and the state of the structure as seen from a desired viewpoint may be drawn. The state of such a structure as seen from the user's side may be superimposed on the image of the virtual world displayed on the HMD 18 so that the correspondence between the angle of view and the virtual world can be seen at a glance.

[0061] Fig. 11 shows another example of an image in which an overhead view is displayed as a user guide. In this example, the direction in which the user is facing is always vertically upward. Therefore, a circle 162 representing the user is displayed in the center of the bottom edge of the image, and a rectangle 160 representing the imaging device is displayed in a position that reflects the relative position with respect to the user. Furthermore, in this figure, information related to the user's field of view is represented by a line 164 instead of information related to the camera's angle of view in Fig. 10.

[0062] Specifically, it shows radial lines that represent the directions that divide the user's effective viewing angle into four equal parts, and concentric arcs that represent the distance from the user at a predetermined interval. Here, the effective viewing angle is a general value that represents the angle at which the brain can accurately recognize an object. With such a user guide, even if the real world is not visible due to the HMD, it is easy to understand what is in the original field of view and where the imaging device is located in the field of view. This allows the user to intuitively determine in what direction and how much movement is required to face the imaging device. Note that the illustrated example is an overhead view based on the user's direction, but it may also be an overhead view with the imaging device fixed on the upper side as in FIG. 10. In addition, an arrow for guiding the user may be further displayed on the images shown in FIGS. 10 and 11.

[0063] FIG. 12 illustrates an image in which a point cloud is displayed as a user guide. The examples illustrated in FIGS. 7, 9, 10, and 11 are planar views of the real world as viewed from above or from the side, but the real world may be viewed from an oblique angle to make it easier to grasp as a three-dimensional space. FIG. 12 illustrates an example of this, in which a point cloud 172 representing a subject is displayed in a three-dimensional space corresponding to the real world together with a figure 170 representing an imaging device, and is drawn as an image. The point cloud 172 represented here is a set of points representing the surface of a subject, obtained by inversely projecting a depth map of the subject, including the user, into a three-dimensional space.

[0064] As described above, the depth map is an image in which the distance in the depth direction is acquired for each pixel based on the parallax of the stereo images and the value is expressed as a pixel value, and is generated by the position and orientation information acquisition unit 80. The user guide generation unit 82 arranges the points constituting the point cloud 172 in a global coordinate system by plotting each pixel of the depth map in a virtual three-dimensional space based on the distance represented by the pixel value. Objects of the imaging device and a plane representing the floor surface are also arranged in the three-dimensional space. Then, by projecting them onto a predetermined screen coordinate system, an image as shown in the figure can be generated.

[0065] The depth map changes according to changes in the position and posture of the user or other subjects, and thus the point cloud 172 also changes. As a result, as in the examples described above, the user's position and movement relative to the imaging device can be objectively grasped. Furthermore, by making it possible to accept user operations that change the position and posture of the screen surface, the user can confirm the relative position from an angle that is easy for the user to understand.

[0066] Furthermore, when creating a depth map by extracting an image from the entire stereo image, subjects other than the user are also drawn at the same time, so it is possible to check whether there are obstacles in the surroundings, whether children or pets have entered the play area, etc. Note that the illustrated example is composed of only a figure 170 representing an imaging device and a point cloud 172, but as in the previous examples, it is also possible to represent the play area or the camera's angle of view, or to show the direction in which the user should move with an arrow.

[0067] Fig. 13 shows another example of an image in which an overhead view is displayed as a user guide. In this example, as in the image of Fig. 7, a rectangle 180 representing an imaging device, a circle 182 representing a user, and an area 184 representing a play area are displayed so as to reflect their actual positional relationship and orientation. Furthermore, in this figure, the presence of objects other than the user in the play area is represented by shapes corresponding to each of them. Specifically, a circle 186 representing an input device and a cross 188 representing an obstacle are displayed.

[0068] The positions of the input device and obstacles are obtained by extracting images other than the image of the user using the depth map or background subtraction method described above. The input device can be distinguished from other objects by the luminous markers attached to the outside or the shape of the input device itself. Objects that are moving other than the user can be extracted by taking the motion difference from the previous frame. With this kind of user guide, even if the user leaves the input device behind or needs the input device during information processing, the user can move to the location of the input device and grasp it without removing the HMD. Even if there are obstacles around or children or pets enter the play area, the danger of collisions can be avoided by taking pre-emptive measures such as recognizing them and moving them away.

[0069] In the example shown in the figure, a figure representing an input device and a figure representing an obstacle are shown at the same time, but of course, either one may be shown depending on the situation. In particular, in dangerous situations such as when an obstacle is detected in the play area, a user guide showing its location in a form may be displayed in a position that is easy for the user to notice, such as the center of the screen, regardless of the user's operation. It is preferable that the figures representing the user, input device, and obstacles are in a form that can be distinguished at a glance. In addition to making the shapes significantly different as shown in the figure, they may also be colored differently or processed in various different ways.

[0070] Also, depending on the situation, the figure may be displayed as an animation that blinks or vibrates slightly. For example, when the user is about to leave the play area or when an obstacle enters the play area, such animation display can alert the user. If the imaging device 12 itself falls or changes direction due to some external force, the rectangle 180 representing the imaging device may be blinked to notify the user. Such a change in the imaging device 12 may be detected by the position and orientation information acquisition unit 80 or the user guide generation unit 82 by the captured image acquisition unit 74 separately acquiring a measurement value by an acceleration sensor built into the imaging device 12.

[0071] Next, the operation of the information processing device 10 that can be realized by the above-mentioned configuration will be described. Fig. 14 is a flowchart showing the processing procedure in which the information processing device generates output data according to the user's movements in this embodiment. This flowchart is started when the user requests the information processing device 10 to start processing via the input device 14.

[0072] First, the captured image acquisition unit 74 of the information processing device 10 requests the imaging device 12 to start capturing images, and in response starts acquiring data of images captured and output by the imaging device 12 (S10). Meanwhile, the user guide generation unit 82 reads out setting information such as the field angle of the camera of the imaging device 12, the corresponding play area, the timing to display the user guide, and the contents to be displayed accordingly from the guide data storage unit 85 (S12). At this time, setting information associated with the content to be implemented, such as that selected by the user, may be selected and read out.

[0073] Next, the position and orientation information acquisition unit 80 acquires information on the user's position and orientation based on the captured image (S14). The information processing unit 76 then uses this information to perform information processing, and the output data generation unit 78 draws a display image of the content as a result (S16). Meanwhile, the user guide generation unit 82 checks whether the situation at that time satisfies the conditions for displaying the user guide (S18). Here, the conditions for displaying the user guide include when a user operation is performed to display the user guide, when there is a danger such as an obstacle in the play area, or when a factor that hinders the normal progress of information processing, such as an abnormality in the orientation of the imaging device 12, is detected.

[0074] The user guide generating unit 82, for example, sets a flag stored in an internal register or the like when such a state occurs, and lowers the flag when such a state is resolved, thereby branching the process conditionally based on the value of the flag. If the conditions for displaying the user guide are not met (N in S18), the output data transmitting unit 86 outputs the image of the content drawn in S16 to the HMD 18 as is (S20). If the conditions for displaying the user guide are met (Y in S18), the user guide generating unit 82 generates an image of the user guide based on the position and orientation information acquired by the position and orientation information acquiring unit 80.

[0075] The data is then supplied to the output data generating unit 78, which then superimposes the image of the user guide on the image of the content (S20). The output data transmitting unit 86 then outputs the image with the superimposed user guide to the HMD 18 (S22). At the same time as the process of S20, audio data of the user guide may also be superimposed on the audio data of the content. For example, specific verbal expressions or instructions such as "Please crouch" or "Please move to the right" may be used, or sound image localization technology may be used, such as generating a predetermined warning sound localized on the boundary line when the user approaches the boundary line of the play area.

[0076] The data output as the user guide may be a combination of images and audio, or just one of them. Depending on the settings, the images captured by the imaging device 12 may be output to the HMD 18 as the user guide as is. This allows the user to determine from the actual image whether there is an obstacle in the field of view of the camera, whether there is an abnormality in the installation of the camera, etc. In this case, the user guide generating unit 82 may superimpose text information such as "Is there anything dangerous?" and a GUI (Graphical User Interface) for the user to input that he or she has confirmed the presence of the object on the captured image, to prompt the user to input the confirmation.

[0077] When there is no need to end the process, such as when the user requests to stop the process, the image of the content is drawn based on the position information, and when the conditions are met, the image is output to the HMD 18 with a user guide superimposed thereon, and this process is repeated at a predetermined rate (N in S24, S14 to S22).When it becomes necessary to end the process, all the processes are ended (Y in S24).

[0078] According to the embodiment described above, in a system that acquires user position information based on an image in a captured image and processes information using the result, information related to the user's surroundings in the real world is presented as a user guide. This allows the user to recognize his / her position and movement direction in the real world even when the user is wearing an HMD and cannot see the outside world, so that the possibility of moving out of the camera's angle of view or the expected play area, causing a malfunction in information processing, or colliding with an object outside the angle of view can be reduced. In addition, by simultaneously presenting the state of objects other than the user as a user guide, collisions in the play area can also be avoided.

[0079] Since the above confirmations can be made without removing the HMD, the burden on the user in recognizing the current situation and avoiding danger is reduced. Furthermore, by overlaying the display in a suitable position on the screen when necessary, the user can check the content while looking at the image displayed on the HMD, without compromising the worldview. Furthermore, things that previously had to be done by feeling, such as stride length and direction of movement, are visualized in correspondence with the position of the imaging device and the range of the play area in the real world, making it easier to learn appropriate movements. In addition, since it uses position and posture information used in the original information processing, it can be achieved with a small processing load.

[0080] In addition, by allowing users to switch the orientation standard when displaying an overhead view, and the viewpoint when displaying a 3D space, users can intuitively determine the direction they should move in, etc. Furthermore, by providing users with movement guidelines using arrows and audio, it is possible to accommodate users of all ages and levels of understanding.

[0081] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0082] For example, in the embodiment, an image that allows the user to understand the positional relationship with respect to the imaging device is presented as a user guide. On the other hand, in particular when the user's position is identified by means other than the imaging device that captures the user, such as a camera attached to an HMD or a motion sensor, the positional relationship with objects other than the imaging device may be displayed. For example, when a play area is set according to some rules, it is sufficient to only show the user's position in the play area. Alternatively, when play equipment or other users are installed in the real space and the positional relationship with them is reflected in information processing, the positions of those objects or people may be displayed in the user guide instead of the imaging device. [Explanation of symbols]

[0083] 8 information processing system, 10 information processing device, 12 imaging device, 14 input device, 16 flat display, 18 HMD, 22 CPU, 24 GPU, 26 main memory, 72 input information acquisition unit, 74 captured image acquisition unit, 76 information processing unit, 78 output data generation unit, 80 position and orientation information acquisition unit, 82 user guide generation unit, 85 guide data storage unit, 86 output data transmission unit.

Claims

1. a position information acquisition unit that acquires position information of a head mounted display; an information processing unit that processes information using the position information; an output data generation unit that generates and outputs data of a display image of a virtual world to be displayed as a result of the information processing; a user guide generating unit for generating data of an image of a user guide that semi-transparently represents a state of the real world in which the user is located; Equipped with The information processing device according to claim 1, wherein the output data generating unit superimposes an image of the user guide on a display image of the virtual world during a period in which a condition for displaying the user guide is satisfied.

2. 2 . The information processing apparatus according to claim 1 , wherein the output data generating section superimposes an image of the user guide when a user operation for displaying the user guide is performed as the condition.

3. The information processing apparatus according to claim 1 , wherein the output data generating unit superimposes the image of the user guide when a result of image recognition using an image of the real world satisfies the condition.

4. The information processing apparatus according to claim 1 , wherein the user guide generating unit generates data of an image of the user guide that represents a three-dimensional space of the real world.

5. 5. The information processing device according to claim 1, wherein the user guide generating section distinguishes between a play area in the real world and other areas in the user guide by using different colors.

6. The information processing device according to claim 1 , wherein the user guide generating unit represents a boundary of a play area in the real world in the user guide.

7. 5. The information processing apparatus according to claim 1, wherein the user guide generating unit represents, in the user guide, how a three-dimensional structure that equally divides a distance in the real world appears as seen from a viewpoint.

8. 5. The information processing apparatus according to claim 1, wherein the user guide generating unit displays, in the user guide, lines that equally divide a distance in a horizontal direction and / or a vertical direction in the real world.

9. the position information acquisition unit generates a depth map in which a pixel value represents a distance from an imaging device that captures the real world to a subject; 9. The information processing apparatus according to claim 1, wherein the user guide generating unit acquires positions of real objects other than the user by using the depth map and displays the positions in the user guide.

10. The information processing device according to any one of claims 1 to 8, characterized in that the user guide generation unit obtains the positions of real objects other than the user based on an image extracted from an image of the real world by a background subtraction method, and displays the positions in the user guide.

11. 11. The information processing apparatus according to claim 1, wherein the user guide generating unit represents, in the user guide, a direction in which the user is actually facing.

12. 12. The information processing device according to claim 1, wherein the output data generating unit blurs the edges of the user guide at a boundary between the display image of the virtual world and the image of the user guide.

13. 13. The information processing apparatus according to claim 1, wherein the position information acquisition unit acquires the position information by an imaging device attached to the head mounted display.

14. acquiring position information of a head mounted display; performing information processing using the position information; generating and outputting data of a display image of a virtual world to be displayed as a result of the information processing; A step of generating data of an image of a user guide that semi-transparently represents a state of the real world in which the user is located; Including, A method for presenting a user guide by an information processing device, characterized in that the output step comprises superimposing an image of the user guide on a display image of the virtual world for a period during which a condition for displaying the user guide is satisfied.

15. A function to obtain position information from the head-mounted display, A function of performing information processing using the position information; A function of generating and outputting data of a display image of a virtual world to be displayed as a result of the information processing; A function to generate user guide image data that shows the state of the real world in which the user is located semi-transparently, This is realized on a computer. A computer program characterized in that the output function superimposes an image of the user guide on a display image of the virtual world during a period in which a condition for displaying the user guide is satisfied.

16. A head mounted display that displays an image generated by an information processing device, A communication unit for transmitting and receiving data between the information processing device and the communication unit; a display unit that displays a display image of a virtual world generated as a result of information processing performed by the information processing device using position information of the head mounted display, and that displays a user guide image, which shows a state of the real world in which the user is located, semi-transparently, superimposed on the display image of the virtual world during a period in which a condition for displaying the user guide is satisfied; A head mounted display comprising:

Images