Information processing device and information processing method

Abstract

To easily and appropriately set a play area where a user wearing a head mount display can move.SOLUTION: An image generation device detects an obstacle present in the surroundings of the device on the basis of an image photographed by a stereo camera of a head mount display, and sets a boundary 72 of a play area where a user wearing the head mount display can move. When a new obstacle is detected inside the play area, the image generation device generates tangential lines 82a and 82b from an observation point H on a floor to a contour 80 of the obstacle, and sets a boundary 86 of a new play area with the tangential lines and a line composed between contact points AB in the contour 80.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus and an information processing method for processing data related to a head-mounted display. [Background technology]

[0002] Image display systems that allow users wearing head-mounted displays to view a target space from any viewpoint are becoming widespread. For example, there are known electronic contents that realize virtual reality (VR) by using a virtual 3D space as the display target and displaying images on the head-mounted display according to the user's gaze direction. Using a head-mounted display can also enhance immersion in images and improve the usability of applications such as games. Furthermore, walk-through systems have been developed that allow users wearing head-mounted displays to virtually walk around the displayed space by physically moving. [Overview of the project] [Problems that the invention aims to solve]

[0003] The range of movement a user wearing a head-mounted display can perform while playing an application needs to be limited according to the size of the user's room and the surrounding environment, such as furniture and belongings. However, there is a dilemma: excessive restrictions can diminish the inherent entertainment value of the content or cause stress to the user.

[0004] This invention was made in view of these problems, and its purpose is to provide a technology that allows a user wearing a head-mounted display to easily and appropriately set up a play area that can be moved. [Means for solving the problem]

[0005] To solve the above problems, one aspect of the present invention relates to an information processing device. This information processing device includes: a camera image acquisition unit that acquires camera image data of the space around the user, captured by a camera mounted on a head-mounted display; a play area setting unit that detects obstacles in the surrounding space based on the camera image and sets a play area in which the user can move based on the result; and a warning processing unit that performs a warning process to the user according to the relationship between the user's position and the boundary of the play area while the user is playing an application using a head-mounted display. The play area setting unit is characterized by identifying the positional relationship between the boundary of the set play area and a newly detected obstacle, and deciding whether or not to modify the boundary of the play area according to the result.

[0006] Another aspect of the present invention relates to an information processing method. This information processing method includes the steps of: acquiring camera image data of the space around the user, captured by a camera mounted on a head-mounted display; detecting obstacles in the surrounding space based on the camera image and setting a play area in which the user can move based on the results; and executing a warning process for the user according to the relationship between the user's position and the boundary of the play area while the user is playing an application using the head-mounted display. The step of setting the play area is characterized by identifying the positional relationship between the boundary of the set play area and a newly detected obstacle, and deciding whether or not to modify the boundary of the play area according to the results.

[0007] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between systems, computer programs, recording media on which computer programs are recorded in a readable manner, data structures, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0008] According to the present invention, a play area that a user wearing a head-mounted display can move around in can be easily and appropriately configured. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the appearance of the head-mounted display in this embodiment. [Figure 2] This figure shows an example of the configuration of the image display system in this embodiment. [Figure 3] This figure illustrates an example of the image world that the image generation device displays on the head-mounted display in this embodiment. [Figure 4] This figure shows the internal circuit configuration of the image generation device in this embodiment. [Figure 5] This figure shows the internal circuit configuration of the head-mounted display in this embodiment. [Figure 6] This is a block diagram showing the functional blocks of the image generation device in this embodiment. [Figure 7] This flowchart shows the operation of the image generation device when setting the play area in this embodiment. [Figure 8] This diagram schematically illustrates the user's actions during the loop processing steps S10 to S14 in Figure 7. [Figure 9] This figure shows an example of a camera image acquired by the camera image acquisition unit during the process of S10 in Figure 7. [Figure 10] This figure shows an example of keyframe data generated from a camera image in Figure 9. [Figure 11] This figure shows the multiple bins that can be used in this embodiment. [Figure 12] This diagram schematically shows how the play area detection unit sets the boundaries of the play area in this embodiment. [Figure 13] This figure illustrates an example of a procedure for correcting a play area when the play area detection unit detects another obstacle inside the set play area in this embodiment. [Figure 14] This diagram schematically shows the modified play area when two new obstacles are detected in this embodiment. [Figure 15] This is a diagram for explaining another example of the procedure for correcting the play area when the play area detection unit detects another obstacle inside the set play area in this embodiment. [Figure 16] This is a flowchart showing the processing procedure for the play area detection unit to detect the play area in this embodiment. [Figure 17] This is a diagram showing an example of the play area editing screen in this embodiment. [Figure 18] This is a flowchart showing the operation of the image generation device during the execution of the VR game in this embodiment.

Embodiments for Carrying Out the Invention

[0010] This embodiment relates to an image display system that displays an application image on a head-mounted display worn on a user's head. FIG. 1 shows an external appearance example of the head-mounted display 100. The head-mounted display 100 of this embodiment is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that wraps around the head when worn by the user to achieve fixation of the device.

[0011] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and a display panel is provided inside so as to face the eyes during wearing. The display panel of the head-mounted display 100 of this embodiment is assumed to be non-transmissive. That is, the head-mounted display 100 is an optically non-transmissive type of head-mounted display.

[0012] The housing 108 may further include an eyepiece positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, to expand the user's field of view. The head-mounted display 100 may also include speakers or earphones positioned to correspond to the user's ears when worn. Furthermore, the head-mounted display 100 incorporates motion sensors to detect the translational and rotational movements of the user's head, as well as its position and orientation at each moment in time.

[0013] Furthermore, the head-mounted display 100 is equipped with a stereo camera 110 on the front of the housing 108. The stereo camera 110 captures video of the surrounding real space with a field of view corresponding to the user's gaze. By displaying the captured images immediately, it is possible to achieve so-called video see-through, where the user can see exactly what is happening in the real space in the direction they are facing. Moreover, by drawing virtual objects on top of the images of real objects captured in the image, augmented reality (AR) can be realized.

[0014] Figure 2 shows an example configuration of the image display system according to this embodiment. The image display system comprises a head-mounted display 100, an image generation device 200, and a controller 140. The head-mounted display 100 is connected to the image generation device 200 via wireless communication. The image generation device 200 may also be connected to a server via a network. In that case, the server may provide the image generation device 200 with data for online applications such as games that multiple users can participate in via the network.

[0015] The image generation device 200 is an information processing device that identifies the position and direction of the viewpoint based on the position and posture of the user's head wearing the head-mounted display 100, generates a display image to match the field of view, and outputs it to the head-mounted display 100. For example, the image generation device 200 may generate a display image of the virtual world, which is the setting of an electronic game, while the game is progressing, or it may display moving images for viewing or information provision, regardless of whether they are virtual or real worlds. Furthermore, by displaying a panoramic image with a wide field of view centered on the user's viewpoint on the head-mounted display 100, it is possible to give the user a deep sense of immersion in the displayed world. The image generation device 200 may be a home game console or a PC.

[0016] The controller 140 is held in the user's hand and is a controller (e.g., a game controller) that receives user input for controlling image generation in the image generation device 200 and image display in the head-mounted display 100. The controller 140 is connected to the image generation device 200 by wireless communication. As a variation, either the head-mounted display 100 or the controller 140, or both, may be connected to the image generation device 200 by wired communication via a signal cable or the like.

[0017] Figure 3 illustrates an example of an image world displayed on a head-mounted display 100 by an image generation device 200. In this example, a state is created in which user 12 is in a virtual room. As shown in the figure, objects such as walls, floors, windows, tables, and objects on the tables are placed in the world coordinate system that defines the virtual space. The image generation device 200 defines a view screen 14 in the world coordinate system according to the position and direction of the user 12's viewpoint, and renders the display image by representing the images of the objects on it.

[0018] The image generation device 200 acquires the position and direction of the user 12's viewpoint (hereinafter, these may be collectively referred to as "viewpoint") from the head-mounted display 100 at a predetermined rate, and changes the position and direction of the view screen 14 accordingly. This allows the image to be displayed on the head-mounted display 100 within the field of view corresponding to the user's viewpoint. Furthermore, the image generation device 200 can generate a stereo image with parallax and display the stereo image in the left and right areas of the display panel of the head-mounted display 100, allowing the user 12 to experience the virtual space in 3D. This allows the user 12 to experience virtual reality as if they were actually in a room in the displayed world.

[0019] Figure 4 shows the internal circuit configuration of the image generation device 200. The image generation device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and main memory 226. These components are interconnected via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228.

[0020] The communication unit 232 includes peripheral device interfaces such as USB and IEEE1394, and network interfaces such as wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive and non-volatile memory. The output unit 236 outputs data to the head-mounted display 100. The input unit 238 receives data input from the head-mounted display 100 and also receives data input from the controller 140. The recording medium drive unit 240 drives removable recording media such as magnetic disks, optical disks, or semiconductor memory.

[0021] The CPU 222 controls the entire image generation device 200 by executing the operating system stored in the memory unit 234. The CPU 222 also executes various programs (e.g., VR game applications) read from the memory unit 234 or a removable recording medium and loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has both geometry engine and rendering processor functions, performing drawing processing according to drawing commands from the CPU 222 and outputting the drawing results to the output unit 236. The main memory 226 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing.

[0022] Figure 5 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU 120, main memory 122, display unit 124, and audio output unit 126. These units are interconnected via a bus 128. An input / output interface 130 is further connected to the bus 128. The input / output interface 130 is connected to a communication unit 132, which includes a wireless communication interface, a motion sensor 134, and a stereo camera 110.

[0023] The CPU 120 processes information acquired from various parts of the head-mounted display 100 via the bus 128, and also supplies display images and audio data acquired from the image generation device 200 to the display unit 124 and the audio output unit 126. The main memory 122 stores the programs and data necessary for processing by the CPU 120.

[0024] The display unit 124 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the head-mounted display 100. The display unit 124 may achieve stereoscopic vision by displaying a pair of stereo images in areas corresponding to the left and right eyes. The display unit 124 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, and which expand the user's field of view.

[0025] The audio output unit 126 consists of speakers or earphones positioned to correspond to the user's ears when the head-mounted display 100 is worn, allowing the user to hear audio. The communication unit 132 is an interface for sending and receiving data with the image generation device 200, and communication is achieved using known wireless communication technologies such as Bluetooth®. The motion sensor 134 includes a gyro sensor and an accelerometer, and acquires the angular velocity and acceleration of the head-mounted display 100.

[0026] As shown in Figure 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from the left and right viewpoints, with a field of view corresponding to the user's viewpoint. The images captured by the stereo camera 110, which show the space around the user, are hereinafter also referred to as "camera images." Camera images can also be described as images that show objects located in the direction of the user's line of sight (typically in front of the user). The measured values ​​from the motion sensor 134 and the data of the images captured by the stereo camera 110 (camera images) are transmitted to the image generation device 200 via the communication unit 132 as needed.

[0027] In this embodiment of the image display system, a play area is defined that specifies the range of the real world in which a user wearing a head-mounted display 100 can move while playing an application. The play area can be described as the area or range within the user's surrounding space (the real world space surrounding the user) in which the user is permitted to move around while viewing virtual reality images (hereinafter also referred to as "VR images"). If the user attempts to deviate from the play area or has deviated from it while playing an application, the image display system provides the user with a warning that serves as an alert or prompts them to return to the play area.

[0028] The above application is a game application that displays VR images on a head-mounted display 100, and will hereinafter also be referred to as a "VR game". For example, the VR game may be a tennis game that displays a VR image of a virtual tennis court and changes the position of the character on the virtual tennis court in accordance with the user's movement (walking, etc.) in the real world.

[0029] In this embodiment, the image generation device 200 automatically detects the play area that a user wearing the head-mounted display 100 can move through while playing an application. At this time, the image generation device 200 determines the range in which the user can move without colliding with objects by identifying the position and shape of surrounding objects in real space captured by the stereo camera 110 of the head-mounted display 100. The image generation device 200 may also accept user input to edit the automatically detected play area and change the shape of the play area according to that input. This allows the user to efficiently set a play area of ​​any shape.

[0030] Figure 6 is a block diagram showing the functional blocks of the image generation device. As described above, the image generation device 200 performs general information processing such as managing the progress of the VR game and communicating with the server, but Figure 6 shows in detail the functional blocks related to setting the play area. Note that at least some of the functions of the image generation device 200 shown in Figure 6 may be implemented on a server connected to the image generation device 200 via a network.

[0031] Furthermore, the multiple functional blocks shown in Figure 6 can be implemented in hardware using the configuration shown in Figure 4, such as the CPU 222, GPU 224, main memory 226, and storage unit 234, and in software using a computer program that implements the functions of the multiple functional blocks. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof, and are not limited to any one of these.

[0032] The image generation device 200 comprises a data processing unit 250 and a data storage unit 252. The data processing unit 250 performs various data processing operations. The data processing unit 250 transmits and receives data to and from the head-mounted display 100 and the controller 140 via the communication unit 232, output unit 236, and input unit 238 shown in Figure 4. The data storage unit 252 stores data that is referenced or updated by the data processing unit 250.

[0033] The data storage unit 252 includes an App storage unit 254, a play area storage unit 256, and a map storage unit 258. The App storage unit 254 stores data for an application that generates VR images (in this embodiment, a VR game). The play area storage unit 256 stores data related to the play area. The data related to the play area includes data indicating the positions of the point cloud that constitutes the boundary of the play area (for example, the coordinate values ​​of each point in the world coordinate system).

[0034] The map storage unit 258 stores map data for estimating the position of the head-mounted display 100 (i.e., the position of the user wearing the head-mounted display 100). The map data in this embodiment is an image (feature points shown in the image) showing the room in which the user plays the VR game, and includes a set of images (feature points shown in the image) whose positional relationships (position and direction) are known. Specifically, the map data includes multiple sets of pairs that associate the position of the head-mounted display 100, the direction of gaze, and keyframes. The map data may also include other items necessary for so-called self-localization.

[0035] A set of keyframes is data that represents the features of the image seen at the same position and viewing direction. A keyframe is an image generated from camera images captured by the stereo camera 110 of the head-mounted display 100, and is an image that contains a predetermined number or more feature points. The number of feature points that should be included in a keyframe may be 24 or more. Feature points may include corners detected by known corner detection methods, or they may be detected based on the brightness gradient. A keyframe can also be described as a collection of partial images extracted from a camera image.

[0036] The data processing unit 250 comprises a system unit 260, an application execution unit 290, and a display control unit 292. The functions of these multiple functional blocks may be implemented in a computer program. The CPU 222 and GPU 224 of the image generation device 200 may perform the functions of the multiple functional blocks by reading the computer program from the storage unit 234 or recording medium into the main memory 226 and executing it.

[0037] The App execution unit 290 reads data for an application selected by the user (in this embodiment, a VR game) from the App storage unit 254 and executes the application selected by the user. The display control unit 292 transmits various image data (e.g., VR images and AR images) generated by the system unit 260 and the App execution unit 290 to the head-mounted display 100 and displays these images on the display unit 124 (display panel) of the head-mounted display 100.

[0038] The system unit 260 performs system processing related to the head-mounted display 100. The system unit 260 provides common services to multiple applications for the head-mounted display 100 (e.g., VR games). The system unit 260 includes a camera image acquisition unit 262, a play area setting unit 264, a map generation unit 274, a notification unit 278, a position estimation unit 280, and a warning processing unit 282.

[0039] The camera image acquisition unit 262 acquires camera images captured by the stereo camera 110 of the head-mounted display 100, which are transmitted from the head-mounted display 100. The play area setting unit 264 performs various processes related to setting the play area. Specifically, the play area setting unit 264 sets the play area based on the camera images acquired by the camera image acquisition unit 262 and the user's operations input via the controller 140. The play area setting unit 264 includes a play area detection unit 266, a floor setting unit 268, and a play area editing unit 270.

[0040] The play area detection unit 266 automatically detects the play area from the space surrounding the user wearing the head-mounted display 100, based on the camera image acquired by the camera image acquisition unit 262. The floor setting unit 268 accepts user input to edit the height of the floor surface detected as a play area by the play area detection unit 266, and changes the height of the floor surface according to that input.

[0041] The play area editing unit 270 receives user input to edit the play area automatically detected by the play area detection unit 266, and changes the shape of the play area according to that input. For example, the play area editing unit 270 shrinks the play area automatically detected by the play area detection unit 266 according to the user's input.

[0042] The map generation unit 274 generates a map for estimating the user's position based on camera images acquired by the camera image acquisition unit 262, in parallel with the play area detection unit 266 detecting the play area. The map generation unit 274 stores the generated map data in the map storage unit 258. The map generation unit 274 terminates map generation when it receives a predetermined number of camera images, which are multiple camera images capturing the user's surroundings from multiple directions.

[0043] The notification unit 278 notifies the App execution unit 290 of the information necessary for the App execution unit 290 to execute the VR game. The position estimation unit 280 estimates the user's position in the real world, or in other words, the user's position in the play area, based on the map data stored in the map storage unit 258 and the camera image acquired by the camera image acquisition unit 262.

[0044] For example, the position estimation unit 280 may compare multiple keyframes included in the map data with camera images and estimate the user's position based on the comparison result and the position and gaze direction of the head-mounted display 100 associated with each keyframe. Alternatively, the position estimation unit 280 may estimate the user's position using known self-localization techniques such as SLAM (Simultaneous Localization and Mapping). The warning processing unit 282 executes a warning process for the user according to the relationship between the play area boundary and the user's position.

[0045] Next, the operation of the image display system realized by the above configuration will be explained. Figure 7 is a flowchart showing the operation of the image generation device 200 when setting the play area. The user can select to initialize or reset the play area in the system settings menu of the head-mounted display 100. If initial initialization or resetting of the play area is selected, the play area setting unit 264 of the image generation device 200 causes the head-mounted display 100 to display a message prompting the user to look around via the display control unit 292.

[0046] In response, when the user wears the head-mounted display 100 on their head and moves around while looking around, the head-mounted display 100 sequentially transmits data from multiple camera images to the image generation device 200. Each of the data from multiple camera images includes sensor data generated at the time of each camera image generation. The sensor data includes measurements from the motion sensor 134, such as the angular velocity and acceleration of the head-mounted display 100. The camera image acquisition unit 262 of the image generation device 200 acquires the camera image data transmitted from the head-mounted display 100 (S10).

[0047] The play area detection unit 266 of the image generation device 200 automatically detects the play area in the space surrounding the user based on the camera image acquired in S10 (S12). Specifically, the play area detection unit 266 may estimate the three-dimensional shape of the user's room using a known method based on the camera image and the sensor data corresponding to the camera image. Based on the estimated three-dimensional shape of the room, the play area detection unit 266 may detect a plane (typically the floor) perpendicular to the direction of gravity indicated by the sensor data, and detect the result of combining multiple detected planes of the same height as the play area. The play area detection unit 266 stores the play area data, including the coordinate values ​​of the point cloud constituting the boundary of the play area, in the play area storage unit 256.

[0048] The play area detection unit 266 detects the height of the floor surface as the play area when a play area is detected. The height of the floor surface may be, for example, the distance between the floor surface and the head-mounted display 100 in the direction of gravity. The play area detection unit 266 stores data indicating the height of the floor surface in the play area storage unit 256. If the position of the head-mounted display 100 is the origin, the height of the floor surface may be minus 1 meter, for example.

[0049] The map generation unit 274 of the image generation device 200 generates a map for estimating the user's position based on the camera image acquired in S10, in parallel with the processing in S12 (S14). The play area detection unit 266 and the map generation unit 274 repeat the processing in S12 and S14 using the new camera image until a predetermined condition is met indicating that sufficient map data for estimating the user's position has been obtained (N in S16). Once sufficient map data has been obtained, the play area detection unit 266 terminates the play area detection process, and the map generation unit 274 terminates the map generation process (Y in S16).

[0050] Next, the floor setting unit 268 of the image generation device 200 generates a floor adjustment screen based on data indicating the height of the floor stored in the play area storage unit 256. The floor adjustment screen may include an AR image in which an object representing the floor (for example, a semi-transparent grid-like object) is superimposed on a camera image acquired by the camera image acquisition unit 262. The floor setting unit 268 displays the floor adjustment screen on the display panel of the head-mounted display 100 via the display control unit 292. The floor setting unit 268 accepts user input to adjust the height of the floor entered on the floor adjustment screen and changes the height of the floor according to the user's input. The floor setting unit 268 stores data indicating the changed floor height in the play area storage unit 256 (S18).

[0051] Next, the play area editing unit 270 of the image generation device 200 generates an image showing the play area automatically detected by the play area detection unit 266. Specifically, the play area editing unit 270 generates a play area editing screen based on the play area data stored in the play area storage unit 256. The play area editing screen includes an AR image in which an object showing the play area is superimposed on a camera image acquired by the camera image acquisition unit 262. The display control unit 292 of the image generation device 200 displays the play area editing screen on the display panel of the head-mounted display 100 (S20).

[0052] The play area editing unit 270 accepts user input for editing the play area entered on the play area editing screen 60 (S22). Specifically, the play area editing unit 270 changes the shape of the play area in response to user input, that is, it enlarges or reduces the play area in response to user input. The play area editing unit 270 updates the play area data stored in the play area storage unit 256 by storing the data of the play area after the shape change (for example, the coordinate values ​​of the point cloud that constitutes the boundary) in the play area storage unit 256.

[0053] Figure 8 schematically illustrates the user's actions during the loop processing S10-S14 in Figure 7. Within room 30, the user moves around while looking around with the head-mounted display 100 on their head. User 32 in Figure 8 indicates the user's position at this time. Note that User 32 in Figure 8 does not represent all the positions the user has moved to, but only the positions where keyframes, which will be explained next, are generated. Arrow 34 indicates the direction of the user's gaze. Play area 36 shows an example of a play area that can be set. As shown in the figure, while the user is moving, the play area setting unit 264 of the image generation device 200 displays the camera image acquired by the camera image acquisition unit 262 on the display panel of the head-mounted display 100 via the display control unit 292, that is, it performs video see-through, showing the user the actual appearance of the real space in the direction the user is facing. This enhances user safety.

[0054] Figure 9 shows an example of a camera image acquired by the camera image acquisition unit 262 in the processing of S10 in Figure 7. Figure 10 shows an example of keyframe data generated from the camera image in Figure 9. The map generation unit 274 extracts multiple feature points 44 contained in the camera image 40 using a known method such as corner detection, and generates a keyframe 42 that shows the extracted multiple feature points 44. If the map generation unit 274 extracts 24 or more feature points 44 from a single camera image 40, it stores the keyframe 42 containing the extracted 24 or more feature points 44 in the map storage unit 258, associating it with the user's position and gaze direction identified by the sensor data.

[0055] The map generation unit 274 also identifies the arrangement and shape of surrounding objects by obtaining the position coordinates in three-dimensional space of feature points extracted from each camera image using well-known techniques such as stereo matching. This three-dimensional spatial information is known as an environmental map. The map generation unit 274 also stores the identified three-dimensional spatial information as map data in the map storage unit 258.

[0056] The map generation unit 274 stores multiple criteria (hereinafter referred to as "bins") for deriving a score regarding the proportion of the user's surrounding space covered by the input camera image (in this embodiment, keyframes generated from the camera image). The multiple bins are multiple criteria that divide the user's surrounding space in different ways.

[0057] Figure 11 shows several bins that can be used in this embodiment. Bin 50, Bin 52, and Bin 54 each divide the space around the user, with the user's position at its center and the direction of gravity as the axis. Bin 50 is a reference that divides the space around the user into four sections with a central angle of 90 degrees, based on the user's line of sight at the start of setting up the play area and the direction perpendicular to the line of sight. Bin 52 is shifted in phase by 45 degrees relative to Bin 50. Bin 54 is a reference that divides the space around the user into sixteen sections with a central angle of 22.5 degrees. Hereinafter, the divided areas in the space around the user defined by Bin 50, Bin 52, and Bin 54 will be referred to as "divided areas".

[0058] The map generation unit 274 derives a score based on bins 50, 52, and 54, relating to the proportion of the user's surrounding space covered by the input camera images (in this embodiment, keyframes generated from camera images). For example, in S16 of Figure 7, if the score exceeds a predetermined threshold, the map generation unit 274 determines that sufficient map data has been obtained and terminates the map generation process.

[0059] Specifically, when the map generation unit 274 generates a keyframe from the camera image, it considers that the division area corresponding to the line of sight obtained from the keyframe has been filled in for each of the bins 50, 52, and 54, and counts the total number of filled division areas in each bin as the score. When the map generation unit 274 generates a new keyframe, it adds the number of newly filled division areas in each bin to the score using the same algorithm.

[0060] For example, the map generation unit 274 determines that there is insufficient map data when the score is less than the threshold "10". As a result, the play area detection process and the map generation process continue. When the score reaches the threshold "10" or higher, the map generation unit 274 determines that sufficient map data has been acquired and terminates the map generation process. This also terminates the play area detection process. The score threshold "10" is a value that can be achieved if the user wearing the head-mounted display 100 looks around 180 degrees, but cannot be achieved if the user does not look around.

[0061] Thus, during the period until it is determined that sufficient map data has been acquired, the play area detection unit 266 gradually sets the boundaries of the play area in parallel with the map generation unit 274 generating map data. Figure 12 schematically shows how the play area detection unit 266 sets the boundaries of the play area. The play area detection unit 266 basically identifies the presence of an object by detecting a surface that is approximately perpendicular to the floor surface in the map generated by the map generation unit 274.

[0062] In the example shown in the figure, as the user wearing the head-mounted display 100 looks around, the three-dimensional position coordinates of feature points of obstacles (e.g., obstacles 70a, 70b) captured in their line of sight (e.g., directions S, S') are gradually identified. As a result, the area of ​​real space where the presence of obstacles is determined expands, and the play area detection unit 266 sets the play area boundary 72 in front of the detected obstacles. In the figure, the play area boundary 72 is represented by a line on the floor, but in reality, it may have a surface perpendicular to the floor.

[0063] Until it is determined that sufficient map data has been obtained based on the criteria using the bins described above, the play area detection unit 266 updates the play area boundary 72 as needed in response to the detection of new obstacles. As shown in the figure, there are cases where an additional obstacle 74 is detected in front of the set play area boundary 72, i.e., inside the play area. For example, this occurs when the height of obstacle 74 is low and obstacle 70b behind it is detected, but obstacle 74 does not come into the field of view of the stereo camera 110.

[0064] At this point, the play area detection unit 266 may consider changing the play area boundary 72 to the boundary 76 in front of the obstacle 74. However, in this case, the play area may become unnecessarily narrow, potentially halving the enjoyment of the game. Therefore, the play area detection unit 266 modifies the play area according to rules that prevent the play area from becoming unnecessarily narrow.

[0065] Specifically, the play area detection unit 266, upon detecting an obstacle 74, limits the area behind the obstacle 74 as seen from the head-mounted display 100, and modifies the play area boundary to exclude that area from the play area. In other words, the play area detection unit 266 ensures that the areas to the left and right (side areas) of the obstacle 74, as seen from the head-mounted display 100, are not excluded from the play area.

[0066] Figure 13 illustrates an example of the procedure for correcting a play area when the play area detection unit 266 detects another obstacle inside the set play area. (a) shows a situation similar to Figure 12, where a new obstacle is detected in front of the provisional play area boundary 72. Here, the contour 80 shows the contour when the three-dimensional shape of the newly detected obstacle is projected onto the floor surface. The play area detection unit 266 generates the contour 80 based on the three-dimensional position coordinates of the feature points of the obstacle.

[0067] The play area detection unit 266 also determines the position where the perpendicular from the center of gravity of the head-mounted display 100 intersects the floor surface when a new obstacle is detected, and defines this position as observation point H. Observation point H corresponds to the user's position on the floor surface. Next, as shown in (b), the play area detection unit 266 generates tangents 82a and 82b from observation point H to the contour 80, and determines the points of contact A and B, as well as the intersection points C and D of the tangents 82a and 82b with the boundary 72.

[0068] Then, as shown in (c), the play area detection unit 266 deletes the portion of the original boundary 72 between intersections C and D, and uses the straight lines AC and BD, shown in thick lines, and the portion of the contour 80 closer to observation point H, between contacts A and B, as the new boundary. As a result, as shown in (d), a new play area boundary 86 is formed, with the newly detected obstacle and the area 84 behind it removed. By modifying the play area in this way, it is possible to avoid collisions with obstacles while allowing the user to move as far as possible around them.

[0069] Although the diagram shows a two-dimensional process on the floor surface, as mentioned above, the boundary of the play area is set in three-dimensional space, including an axis perpendicular to the floor surface. In other words, the boundary 86 of the new play area can actually be composed of a plane perpendicular to the floor surface. In this case, the observation point H is the position of the head-mounted display 100 or the user when an obstacle is detected, and the tangents 82a and 82b to the contour 80 can be said to be two vertical planes that are in contact with the obstacle. Furthermore, the portion of the contour 80 between contact points A and B can be said to be a third vertical plane that is externally tangent to the obstacle and intersects with the two vertical planes on the user side. The boundaries shown in Figures 14 and 15 are similar.

[0070] Figure 14 schematically shows the modified play area when two new obstacles are detected. The play area detection unit 266 independently applies the same procedure as in Figure 13 to modify the play area boundary for each obstacle 300a and 300b. The resulting new boundary 301 allows the area between obstacles 300a and 300b to remain as a play area as much as possible. As a result, collisions with these obstacles can be avoided while minimizing the reduction in the play area. The same applies when three or more obstacles are detected.

[0071] According to the play area boundary modification procedure shown in Figure 13, the shape of the modified boundary depends on the position of the head-mounted display 100 when a new obstacle is detected. Qualitatively, the further the head-mounted display 100 is from the obstacle when it is detected, the narrower the apparent width of the obstacle becomes, and therefore the narrower the width of the play area that is deleted can be.

[0072] For example, in games where the user does not move dynamically, it can be expected that a certain distance will be maintained between the head-mounted display 100 and the obstacle during gameplay. Therefore, it is reasonable to limit the reduction in the play area for obstacles detected at a distance, as leaving a somewhat wider play area around the obstacle reduces the likelihood of the user deviating from the play area and colliding with the obstacle.

[0073] On the other hand, if an obstacle is detected at close range and its apparent width increases, there is a risk of unnecessarily excluding the area behind the obstacle from the play area. Therefore, the play area detection unit 266 checks the distance to the head-mounted display 100 when an obstacle is detected using a predetermined indicator and switches the procedure for correcting the play area boundary. This avoids a situation where the play area is reduced unnecessarily depending on the position of the head-mounted display 100 when an obstacle is detected, and ensures that the play area is stably secured.

[0074] Figure 15 illustrates another example of the procedure for correcting the play area when the play area detection unit 266 detects another obstacle inside the set play area. The diagram is presented in the same way as in Figure 13, with (a) showing a situation where a new obstacle has been detected in front of the provisional play area boundary 72. The contour 302 is the contour of the newly detected obstacle when its three-dimensional shape is projected onto the floor, and observation point H represents the position where the perpendicular from the center of gravity of the head-mounted display 100 intersects the floor when the new obstacle is detected.

[0075] The play area detection unit 266 evaluates the distance to the head-mounted display 100 when it detects a new obstacle using a predetermined index, and if it determines that the distance is closer than the predetermined standard, it adopts the correction procedure shown in the figure. In the figure, tangents 304a and 304b are generated from the observation point H to the contour 302, and the angle θ is made between either of these and the center line 308 passing through the center point 306 of the internal region of the contour 302 from the observation point H. o This is used as an indicator of the distance between the head-mounted display 100 and the obstacle.

[0076] Naturally, the angle θ o A larger value indicates a shorter distance. This index also takes into account the actual size of the obstacle, and from an overall perspective, it can be seen as an indicator of the likelihood that the play area will be reduced more than necessary. However, the index used is not limited to this; the distance between observation point H and contour 302 may also be used. Angle θ o When using an index, for example, the angle θo is greater than a threshold θ such as 45° th When it is greater, the play area detection unit 266 determines that the distance between the head-mounted display 100 and the obstacle is short, and thus it is highly likely that the play area is excessively reduced by the method of FIG. 13.

[0077] Note that for the angle θ o an upper threshold greater than the threshold θ th may also be set. That is, when the angle θ o is close to 180°, it is conceivable that the observation point H is in contact with the contour 302 or overlaps with the internal area of the contour 302. In this case, the contour 302 is not that of a true obstacle and is highly likely to be derived from the user's own body. Therefore, when the angle θ o is greater than a predetermined threshold close to 180°, the play area detection unit 266 does not correct the play area boundary. When using the distance between the observation point H and the contour 302 as an index, when the distance is less than or equal to a threshold such as 0, the play area detection unit 266 does not correct the play area boundary.

[0078] The angle θ o is less than or equal to the upper threshold and greater than a threshold θ for switching the correction method such as 45° th In this case, as shown in (b), the play area detection unit 266 generates new tangents 312a and 312b whose included angle 2θ o ’ is less than the angle 2θ o . The tangents 312a and 312b are lines obtained by translating lines 310a and 310b that are symmetric with respect to the center line 308 passing through the observation point H and form an angle θ o ’ (<θ o ) with the center line 308 so as to be tangent to the contour 302.

[0079] The angle θ o' is, for example, 25°. The play area detection unit 266 then finds the points of contact A' and B' between the new tangents 312a and 312b and the contour 302, and the intersection points C' and D' between the tangents 312a and 312b and the boundary 72. As shown in (c), the play area detection unit 266 then deletes the portion of the original boundary 72 between the intersection points C' and D', and sets the straight lines A'C' and B' and the portion of the contour 302 closer to the observation point H between the points of contact A' and B' as the new boundary.

[0080] As a result, as shown in (d), a new play area boundary 316 is formed, with the newly detected obstacle and the area 314 behind it removed. This reduces the amount of play area reduction compared to modifying the play area using the original tangents 304a and 304b. Note that the illustrated procedure is just one example, and the line tangent to the contour 302 may be a polyline or a curve, for example. However, the illustrated method allows for the reliable removal of obstacles from the play area with simple calculations.

[0081] Figure 16 is a flowchart showing the processing procedure by which the play area detection unit 266 detects the play area. This process corresponds to S12 in Figure 7. Based on the feature points extracted from the camera image, the play area detection unit 266 determines whether a new obstacle or a new surface of an already detected obstacle has been detected (S30). If nothing is detected, the play area detection unit 266 terminates the process (N in S30). If a new obstacle or surface is detected (Y in S30), the play area detection unit 266 determines whether the detected obstacle or surface is in front of the provisionally set play area boundary, i.e., inside the play area (S32).

[0082] In detail, the play area detection unit 266 checks whether the obstacle detected in S30 is located between the head-mounted display 100 and the provisionally set play area boundary. If such a positional relationship does not exist (N in S32), the play area detection unit 266 assumes that the play area boundary in that direction has not been set and sets the play area boundary in front of the obstacle, ending the process (S36). Although not shown in the diagram, if the detected obstacle is beyond the already set play area boundary, the play area detection unit 266 may simply end the process.

[0083] When the detected obstacle or surface is inside the play area (Y in S32), the play area detection unit 266 determines whether the distance between the head-mounted display 100 and the obstacle at the time the obstacle was detected in S30 is greater than or equal to a standard (S34). In the example in Figure 15, the angle θ o This is used as an indicator, and that is the threshold θ th The distance is determined to be greater than the standard when the following conditions are met: If the distance between the head-mounted display 100 and the obstacle is greater than the standard (Y in S34), the play area detection unit 266 corrects the play area boundary using the tangent line of the obstacle's outline that passes through observation point H, as shown in Figure 13 (S38).

[0084] Specifically, a new boundary is generated using the portion of the obstacle's contour between contact points and a portion of the tangent line with the contact point as the endpoint. If the distance between the head-mounted display 100 and the obstacle is smaller than the reference (N in S34), the play area detection unit 266 corrects the play area boundary using a tangent line of the obstacle's contour that has a narrower angle than the tangent line passing through observation point H, as shown in Figure 15 (S40).

[0085] Specifically, the play area detection unit 266 generates two lines that are symmetrical to the center line of the obstacle's contour, with an angle smaller than the tangent line in S38, and moves them parallel to the contour until they are tangent to it. Then, it generates a new boundary using the portion of the obstacle's contour between the points of contact and a portion of the tangent line with the points of contact as its endpoints. By repeating the illustrated process each time a new obstacle is detected, the boundary of the play area can ultimately be completed. As described above, it is desirable that the play area set by the play area detection unit 266 be adjustable by the user.

[0086] Figure 17 shows an example of a play area editing screen. The play area editing screen 60 includes a play area 62 and a boundary 64. The play area 62 is an image representing the play area (typically an unobstructed floor surface), and may also be an image representing, for example, a semi-transparent grid-like object. The boundary 64 is an image representing the boundary of the play area 62, and is an image that intersects the play area perpendicularly at the boundary of the play area 62. The boundary 64 may also be, for example, a semi-transparent grid-like object. As described above, the play area editing unit 270 accepts user operations to deform, enlarge, or reduce the displayed play area, and stores the edited play area data in the play area storage unit 256.

[0087] Figure 18 is a flowchart illustrating the operation of the image generation device 200 during VR game execution. This flowchart is initiated when the user puts on the head-mounted display 100 and initiates a VR game startup operation on the image generation device 200. In response, the notification unit 278 of the image generation device 200 transmits data of the play area stored in the play area storage unit 256, such as data indicating the shape and size of the play area, to the App execution unit 290 (S50).

[0088] The App execution unit 290 then reads the VR game program data from the App storage unit 254 and starts the VR game (S52). For example, the App execution unit 290 may allow the user to acquire game items by placing them within the play area in a virtual space that depicts the game world. The display control unit 292 displays the VR game image (e.g., VR image) generated by the App execution unit 290 on the display panel of the head-mounted display 100.

[0089] The camera image acquisition unit 262 of the image generation device 200 sequentially acquires camera images transmitted sequentially from the head-mounted display 100, and the position estimation unit 280 sequentially estimates the user's position and gaze direction in the real world (S54). The notification unit 278 notifies the App execution unit 290 of the user's position and gaze direction estimated by the position estimation unit 280 (S56). The App execution unit 290 proceeds with the VR game according to the estimated user's position and gaze direction (S58). For example, the App execution unit 290 may move the user character in the VR game according to changes in the user's position in the real world (in other words, within the play area).

[0090] The warning processing unit 282 of the image generation device 200 detects when the user's position in the real world approaches the boundary of the play area, for example, when the distance from the head-mounted display 100 worn by the user to the boundary of the play area falls below a predetermined threshold (e.g., 30 centimeters) (Y in S60). At this time, the warning processing unit 282 executes a predetermined warning process for the user (S62).

[0091] For example, if the user's position reaches the vicinity of the play area boundary, the warning processing unit 282 may pass an image indicating the play area boundary (for example, the boundary 64 shown in Figure 17) to the display processing unit 292. The display processing unit 292 may superimpose the image indicating the play area boundary onto the game image generated by the App execution unit 290 and display it as a display image on the display panel of the head-mounted display 100. Alternatively, if the user's position in the real world reaches the vicinity of the play area boundary or crosses the play area boundary, the warning processing unit 282 may display a video see-through image on the display panel of the head-mounted display 100 via the display processing unit 292.

[0092] Furthermore, the warning processing unit 282 may first display an image indicating the boundary of the play area when the user's position in the real world reaches the vicinity of the play area boundary (e.g., 30 centimeters), and may then display a video see-through image when the user's position in the real world reaches even closer to the play area boundary (e.g., 10 centimeters). On the other hand, if the user's position in the real world is not near the boundary of the play area (N in S60), S62 is skipped. If a predetermined termination condition is met, such as when the user stops the execution of the VR game (Y in S64), the flow in this diagram is terminated. If the termination condition is not met (N in S64), the process returns to S54 and the VR game continues.

[0093] According to the embodiment described above, the image generation device sets a play area in which a user wearing a head-mounted display can move around safely. Specifically, the image generation device detects obstacles based on images captured by a stereo camera mounted on the head-mounted display and sets the boundaries of the play area to exclude them. In this case, if a new obstacle is detected inside the play area that has already been set, the image generation device sets a new play area boundary using the tangent line from the head-mounted display to the outline of the obstacle at the time of detection.

[0094] This allows the minimum necessary areas—obstacles and the area behind them from the perspective of the head-mounted display—to be excluded from the play area. Since the position of the head-mounted display at the time of detection is estimated to be close to the user's position when playing content such as games, by leaving the area in front of the obstacle as part of the play area and excluding the area behind it, collisions with obstacles can be avoided while minimizing the actual reduction in the play area.

[0095] Furthermore, if the system determines that the distance between the detected obstacle and the head-mounted display is close, the angle formed by the tangent is narrowed to suppress the expansion of the area excluded from the play area. This minimizes the influence of the head-mounted display's position on the size of the play area when obstacles are detected, making it easy to set a sufficient play area regardless of the situation.

[0096] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of Symbols]

[0097] 100 Head-mounted display, 200 Image generation device, 256 Play area storage unit, 264 Play area setting unit, 266 Play area detection unit, 270 Play area editing unit, 274 Map generation unit, 280 Position estimation unit, 282 Warning processing unit, 292 Display control unit.

Claims

1. A camera image acquisition unit that acquires camera image data of the user's surrounding space captured by a camera mounted on a head-mounted display, A play area setting unit detects obstacles in the surrounding space based on the camera image and sets a play area where the user can move based on the result, A warning processing unit that executes a warning process for the user according to the relationship between the user's position while playing the application using the head-mounted display and the boundary of the play area, Equipped with, The information processing device is characterized in that the play area setting unit identifies the positional relationship between the set play area boundary and a newly detected obstacle, determines whether or not to modify the play area boundary according to the result, and, when the obstacle is inside the play area, limits the area behind the obstacle as seen from the head-mounted display at the time of detection and modifies the boundary so as to exclude it from the play area.

2. The information processing device according to claim 1, wherein the play area setting unit, on the floor surface, uses a line formed by the portion from the point of contact with the obstacle to the intersection with the boundary of the play area, and the portion of the contour closer to the head-mounted display, between each point of contact, as the modified boundary.

3. The information processing device according to claim 1 or 2, wherein the play area setting unit confirms the distance between the obstacle and the head-mounted display at the time of detection using a predetermined indicator, and switches the boundary correction method according to the result.

4. The information processing device according to claim 2, wherein the play area setting unit confirms the distance between the obstacle and the head-mounted display at the time of detection using a predetermined index, and when it determines that the distance is closer than the standard, it determines the corrected boundary using two other tangents that form an angle smaller than the angle formed by the two tangents passing through the position of the head-mounted display.

5. The information processing apparatus according to claim 4, characterized in that the play area setting unit generates the other two tangent lines by translating two lines that pass through the position of the head-mounted display at the time of detection and form a predetermined angle symmetrical to the center line of the contour so as to be tangent to the contour.

6. The information processing device according to any one of claims 3 to 5, wherein the play area setting unit determines the proximity of the distance using the magnitude of the angle between two tangents that pass through the position of the head-mounted display at the time of detection and touch the contour of the obstacle as an indicator.

7. The information processing device according to any one of claims 1 to 6, characterized in that when there are multiple obstacles inside the play area, the play area setting unit independently modifies the boundary so as to limit each obstacle and the area behind it and exclude them from the play area.

8. The process involves acquiring camera image data of the user's surroundings, captured by a camera mounted on a head-mounted display, and The steps include detecting obstacles in the surrounding space based on the camera image and setting a play area where the user can move based on the result, The steps include: executing a warning process for the user based on the relationship between the user's position while playing the application using the head-mounted display and the boundary of the play area; Includes, The information processing method is characterized in that the step of setting the play area involves identifying the positional relationship between the boundary of the set play area and a newly detected obstacle, deciding whether or not to modify the boundary of the play area according to the result, and, when the obstacle is inside the play area, limiting the area behind the obstacle as seen from the head-mounted display at the time of detection, and modifying the boundary so as to exclude it from the play area.

9. The head-mounted display has a camera that captures camera image data of the user's surroundings, and the camera data is acquired from the camera that the head-mounted display is equipped with. A function that detects obstacles in the surrounding space based on the camera image and sets a play area that the user can move around in based on the result, A function that executes a warning process for the user based on the relationship between the user's position while playing the application using the head-mounted display and the boundary of the play area, To make this a reality on a computer, The function for setting the play area is a computer program characterized by identifying the positional relationship between the boundary of the set play area and a newly detected obstacle, deciding whether or not to modify the boundary of the play area according to the result, and, when the obstacle is inside the play area, limiting the area behind the obstacle as seen from the head-mounted display at the time of detection, and modifying the boundary so as to exclude it from the play area.

Images