Information processing device, information processing method

Abstract

To easily and appropriately set a play area where a user wearing a head mount display can move.SOLUTION: An image generation device detects an obstacle present in the surroundings of the device on the basis of an image photographed by a stereo camera of a head mount display, and sets a boundary 72 of a play area where a user wearing the head mount display can move. When a new obstacle is detected inside the play area, the image generation device generates tangential lines 82a and 82b from an observation point H on a floor to a contour 80 of the obstacle, and sets a boundary 86 of a new play area with the tangential lines and a line composed between contact points AB in the contour 80.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method for processing data related to a head-mounted display. [Background technology]

[0002] Image display systems that allow users wearing head-mounted displays to view a target space from any viewpoint are becoming widespread. For example, there is electronic content that realizes virtual reality (VR) by displaying a virtual three-dimensional space on the head-mounted display according to the user's line of sight. The use of head-mounted displays can also enhance the sense of immersion in the video and improve the operability of applications such as games. In addition, walk-through systems have been developed that allow users wearing head-mounted displays to virtually walk around a space displayed as a video by physically moving around. Summary of the Invention [Problem to be solved by the invention]

[0003] The range of movement that a user wearing a head-mounted display can make while playing an application needs to be limited depending on the size of the user's room and the environment, such as the furniture and luggage installed in the user's room. On the other hand, excessive restrictions can lead to a dilemma, as they can diminish the entertainment value that the content originally offers and cause stress to the user.

[0004] The present invention has been made in consideration of these problems, and its purpose is to provide a technology that allows a user wearing a head-mounted display to easily and appropriately set a play area that can be moved around. [Means for solving the problem]

[0005] To solve the above problems, one aspect of the present invention relates to an information processing device, which includes: a camera image acquisition unit that acquires data of a camera image of a space surrounding a user captured by a camera mounted on a head-mounted display; a play area setting unit that detects obstacles in the space surrounding the user based on the camera image and sets a play area within which the user can move based on the detected obstacles; and a warning processing unit that executes a warning process for the user based on the relationship between the user's position while playing an application using the head-mounted display and a boundary of the play area, wherein the play area setting unit identifies the positional relationship between the boundary of the set play area and a newly detected obstacle and determines whether to modify the boundary of the play area based on the result of the detection.

[0006] Another aspect of the present invention relates to an information processing method, the information processing method including the steps of: acquiring data of a camera image of a space surrounding a user, the camera image being captured by a camera mounted on a head-mounted display; detecting obstacles in the space surrounding a user based on the camera image and setting a play area within which the user can move based on the detection result; and executing a warning process for the user based on a relationship between the position of the user playing an application using the head-mounted display and a boundary of the play area, wherein the step of setting the play area includes identifying a positional relationship between the boundary of the set play area and a newly detected obstacle and determining whether to modify the boundary of the play area based on the detection result.

[0007] Any combination of the above components, or conversion of the present invention between a system, a computer program, a recording medium on which a computer program is readably recorded, a data structure, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0008] According to the present invention, a play area in which a user wearing a head-mounted display can move can be easily and appropriately set. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing an example of the appearance of a head-mounted display according to an embodiment of the present invention; [Figure 2] 1 is a diagram illustrating an example of the configuration of an image display system according to an embodiment of the present invention. [Figure 3] 1 is a diagram for explaining an example of an image world that the image generating device displays on a head-mounted display in this embodiment. FIG. [Figure 4] FIG. 2 is a diagram showing an internal circuit configuration of the image generating device according to the present embodiment. [Figure 5] FIG. 2 is a diagram showing an internal circuit configuration of a head-mounted display according to the present embodiment. [Figure 6] 1 is a block diagram showing functional blocks of an image generating device according to an embodiment of the present invention. [Figure 7] 10 is a flowchart showing the operation of the image generating device when setting a play area in the present embodiment. [Figure 8] FIG. 8 is a diagram schematically showing user actions in the loop process of S10 to S14 in FIG. [Figure 9] 8 is a diagram showing an example of a camera image acquired by a camera image acquisition unit in the process of S10 in FIG. 7. FIG. [Figure 10] 10 is a diagram showing an example of data of a key frame generated from the camera image of FIG. 9. FIG. [Figure 11] FIG. 10 is a diagram showing a plurality of bins that can be employed in the present embodiment. [Figure 12] 10 is a diagram showing a schematic diagram of how the play area detection unit sets the boundaries of the play area in the present embodiment. FIG. [Figure 13] 10 is a diagram illustrating an example of a procedure for correcting a play area when the play area detection unit detects another obstacle within the set play area in this embodiment. FIG. [Figure 14] FIG. 10 is a diagram schematically showing the revised play area when two new obstacles are detected in this embodiment. [Figure 15] A figure for explaining another example of the procedure for correcting the play area when the play area detection unit detects another obstacle inside the set play area in this embodiment. [Figure 16] 10 is a flowchart showing a processing procedure in which a play area detection unit detects a play area in the present embodiment. [Figure 17] FIG. 10 is a diagram showing an example of a play area editing screen in the present embodiment. [Figure 18] 10 is a flowchart showing the operation of the image generation device when a VR game is executed in the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] This embodiment relates to an image display system that displays application images on a head-mounted display worn on a user's head. Fig. 1 shows an example of the appearance of a head-mounted display 100. The head-mounted display 100 of this embodiment is composed of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that, when worn by the user, wraps around the head and secures the device in place.

[0011] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and is provided with a display panel inside so that it faces the eyes when worn. The display panel of the head-mounted display 100 of this embodiment is not transparent. In other words, the head-mounted display 100 is a light-opaque head-mounted display.

[0012] The housing 108 may further include an eyepiece lens located between the display panel and the user's eyes when the head mounted display 100 is worn, to expand the user's field of view. The head mounted display 100 may further include speakers or earphones at positions corresponding to the user's ears when worn. The head mounted display 100 also includes a built-in motion sensor that detects the translational and rotational movements of the head of the user wearing the head mounted display 100, as well as the position and posture at each time.

[0013] The head-mounted display 100 also includes a stereo camera 110 on the front surface of the housing 108. The stereo camera 110 captures video of the surrounding real space in a field of view that corresponds to the user's line of sight. If the captured image is displayed immediately, the real space in the direction the user is facing can be seen as it is, which is known as video see-through. Furthermore, if a virtual object is drawn on the image of a real object shown in the captured image, augmented reality (AR) can be realized.

[0014] 2 shows an example of the configuration of an image display system according to this embodiment. The image display system includes a head-mounted display 100, an image generation device 200, and a controller 140. The head-mounted display 100 is connected to the image generation device 200 via wireless communication. The image generation device 200 may further be connected to a server via a network. In this case, the server may provide the image generation device 200 with data for an online application such as a game in which multiple users can participate via the network.

[0015] The image generation device 200 is an information processing device that identifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of a user wearing the head-mounted display 100, generates a display image so as to provide a corresponding field of view, and outputs the image to the head-mounted display 100. For example, the image generation device 200 may generate a display image of a virtual world in which the game is set as the electronic game progresses, or may display moving images for viewing or providing information, regardless of whether the world is a virtual world or a real world. Furthermore, by displaying a panoramic image with a wide angle of view centered on the user's viewpoint on the head-mounted display 100, the user can be given a deep sense of immersion in the displayed world. The image generation device 200 may be a stationary game console or a PC.

[0016] The controller 140 is a controller (e.g., a game controller) that is held by the user's hand and into which the user's operations are input to control image generation in the image generation device 200 and image display in the head mounted display 100. The controller 140 is connected to the image generation device 200 by wireless communication. As a variation, one or both of the head mounted display 100 and the controller 140 may be connected to the image generation device 200 by wired communication via a signal cable or the like.

[0017] FIG. 3 is a diagram illustrating an example of an image world that the image generation device 200 displays on the head-mounted display 100. In this example, a state is created in which the user 12 is in a room, which is a virtual space. As shown in the figure, objects such as walls, floors, windows, a table, and objects on the table are arranged in a world coordinate system that defines the virtual space. The image generation device 200 defines a view screen 14 in the world coordinate system according to the position of the viewpoint and the direction of the line of sight of the user 12, and draws a display image by representing the images of objects on the view screen.

[0018] The image generation device 200 acquires the position of the viewpoint and the direction of the line of sight of the user 12 (hereinafter, these may be collectively referred to as the "viewpoint") from the head-mounted display 100 at a predetermined rate, and changes the position and direction of the view screen 14 accordingly. This allows an image to be displayed on the head-mounted display 100 in a field of view corresponding to the user's viewpoint. The image generation device 200 can also generate stereo images with parallax and display the stereo images in the left and right regions of the display panel of the head-mounted display 100, allowing the user 12 to view a virtual space in three dimensions. This allows the user 12 to experience virtual reality as if they were in a room in the displayed world.

[0019] 4 shows the internal circuit configuration of the image generation device 200. The image generation device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and a main memory 226. These components are connected to one another via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228.

[0020] The communication unit 232 includes a peripheral device interface such as USB or IEEE1394, and a network interface such as a wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive, a nonvolatile memory, etc. The output unit 236 outputs data to the head mounted display 100. The input unit 238 accepts data input from the head mounted display 100 and also accepts data input from the controller 140. The recording medium drive unit 240 drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

[0021] The CPU 222 controls the entire image generating device 200 by executing an operating system stored in the storage unit 234. The CPU 222 also executes various programs (e.g., VR game applications) that are read from the storage unit 234 or a removable recording medium and loaded into the main memory 226, or that are downloaded via the communication unit 232. The GPU 224 has the functions of a geometry engine and a rendering processor, performs drawing processing in accordance with drawing commands from the CPU 222, and outputs the drawing results to the output unit 236. The main memory 226 is composed of RAM (Random Access Memory), and stores programs and data necessary for processing.

[0022] 5 shows the internal circuit configuration of the head mounted display 100. The head mounted display 100 includes a CPU 120, a main memory 122, a display unit 124, and an audio output unit 126. These units are connected to one another via a bus 128. An input / output interface 130 is further connected to the bus 128. To the input / output interface 130, a communication unit 132 including a wireless communication interface, a motion sensor 134, and the stereo camera 110 are connected.

[0023] The CPU 120 processes information acquired from each unit of the head mounted display 100 via the bus 128, and supplies display image and audio data acquired from the image generating device 200 to the display unit 124 and audio output unit 126. The main memory 122 stores programs and data necessary for processing by the CPU 120.

[0024] The display unit 124 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of a user wearing the head mounted display 100. The display unit 124 may achieve stereoscopic vision by displaying a pair of stereo images in areas corresponding to the left and right eyes. The display unit 124 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the head mounted display 100 is worn, and that expand the user's field of view.

[0025] The audio output unit 126 is composed of speakers or earphones provided at positions corresponding to the user's ears when the head mounted display 100 is worn, and allows the user to hear audio. The communication unit 132 is an interface for sending and receiving data to and from the image generation device 200, and realizes communication using known wireless communication technology such as Bluetooth (registered trademark). The motion sensor 134 includes a gyro sensor and an acceleration sensor, and acquires the angular velocity and acceleration of the head mounted display 100.

[0026] As shown in FIG. 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints in a field of view corresponding to the user's viewpoint. An image captured by the stereo camera 110, which captures the space around the user, is also referred to as a "camera image" hereinafter. The camera image can also be said to be an image that captures an object that exists in the user's line of sight (typically in front of the user). Measurement values ​​by the motion sensor 134 and data on the images captured by the stereo camera 110 (camera images) are transmitted to the image generating device 200 via the communication unit 132 as necessary.

[0027] In the image display system of this embodiment, a play area is set, which defines the range of the real world within which a user wearing the head-mounted display 100 can move while playing an application. The play area can be said to be an area or range within the user's surrounding space (the real-world space surrounding the user) within which the user is permitted to move while viewing virtual reality images (hereinafter also referred to as "VR images"). If the user attempts to deviate from the play area while playing an application, or if the user deviates from the play area, the image display system provides the user with a warning to warn them or to urge them to return to the play area.

[0028] The above application is a game application that displays a VR image on the head-mounted display 100, and is hereinafter also referred to as a "VR game." For example, the VR game may be a tennis game that displays a VR image showing a tennis court in a virtual world, and changes the position of a character on the tennis court in the virtual world as the user moves (walks, etc.) in the real world.

[0029] In this embodiment, the image generation device 200 automatically detects a play area within which a user wearing the head-mounted display 100 can move while playing an application. In this case, the image generation device 200 identifies the positions and shapes of surrounding objects in real space captured by the stereo camera 110 of the head-mounted display 100, and determines the range within which the user can move without colliding with those objects. The image generation device 200 may further receive a user operation to edit the automatically detected play area, and change the shape of the play area in accordance with the operation. This allows the user to efficiently set a play area of ​​any shape.

[0030] Fig. 6 is a block diagram showing functional blocks of the image generation device. As described above, the image generation device 200 performs general information processing such as progressing through the VR game and communicating with the server, but Fig. 6 particularly shows in detail functional blocks related to setting the play area. Note that at least some of the functions of the image generation device 200 shown in Fig. 6 may be implemented in a server connected to the image generation device 200 via a network.

[0031] 6 can be realized in hardware by the configuration of the CPU 222, GPU 224, main memory 226, storage unit 234, etc. shown in Fig. 4, and can be realized in software by a computer program that implements the functions of the multiple functional blocks. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof, and are not limited to any one of them.

[0032] The image generating device 200 includes a data processing unit 250 and a data storage unit 252. The data processing unit 250 executes various types of data processing. The data processing unit 250 transmits and receives data to and from the head mounted display 100 and the controller 140 via the communication unit 232, output unit 236, and input unit 238 shown in FIG. 4. The data storage unit 252 stores data to be referenced or updated by the data processing unit 250.

[0033] The data storage unit 252 includes an app storage unit 254, a play area storage unit 256, and a map storage unit 258. The app storage unit 254 stores data of an application (in this embodiment, a VR game) that generates VR images. The play area storage unit 256 stores data related to the play area. The data related to the play area includes data indicating the positions of a group of points that form the boundary of the play area (for example, the coordinate values ​​of each point in the world coordinate system).

[0034] The map storage unit 258 stores map data for estimating the position of the head mounted display 100 (i.e., the position of the user wearing the head mounted display 100). The map data in this embodiment is an image (feature points shown in the image) showing the room in which the user is playing the VR game, and includes a collection of images (feature points shown in the image) whose positional relationships (positions and directions) are known. Specifically, the map data includes multiple pairs of associations between the position, line of sight direction, and key frames of the head mounted display 100. Note that the map data may also include other items necessary for so-called self-position estimation.

[0035] A set of key frames is data indicating the characteristics of an image seen at the same set of positions and gaze directions. A key frame is an image generated based on camera images captured by the stereo camera 110 of the head-mounted display 100, and is an image that includes a predetermined number of feature points or more. The number of feature points to be included in a key frame may be 24 or more. The feature points may include corners detected by a known corner detection method, or may be detected based on brightness gradients. A key frame can also be considered a collection of partial images cut out from a camera image.

[0036] The data processing unit 250 includes a system unit 260, an app execution unit 290, and a display control unit 292. The functions of these functional blocks may be implemented in a computer program. The CPU 222 and GPU 224 of the image generating device 200 may perform the functions of the functional blocks by reading the computer program from the storage unit 234 or a recording medium into the main memory 226 and executing it.

[0037] The App execution unit 290 reads data of an application (a VR game in this embodiment) selected by the user from the App storage unit 254, and executes the application selected by the user. The display control unit 292 transmits data of various images (for example, VR images and AR images) generated by the system unit 260 and the App execution unit 290 to the head mounted display 100, and displays these images on the display unit 124 (display panel) of the head mounted display 100.

[0038] The system unit 260 executes system processing related to the head mounted display 100. The system unit 260 provides a common service for a plurality of applications (e.g., VR games) for the head mounted display 100. The system unit 260 includes a camera image acquisition unit 262, a play area setting unit 264, a map generation unit 274, a notification unit 278, a position estimation unit 280, and a warning processing unit 282.

[0039] The camera image acquisition unit 262 acquires camera images captured by the stereo camera 110 of the head mounted display 100 and transmitted from the head mounted display 100. The play area setting unit 264 executes various processes related to setting the play area. Specifically, the play area setting unit 264 sets the play area based on the camera images acquired by the camera image acquisition unit 262 and user operations input via the controller 140. The play area setting unit 264 includes a play area detection unit 266, a floor setting unit 268, and a play area editing unit 270.

[0040] The play area detection unit 266 automatically detects a play area from the surrounding space of the user wearing the head mounted display 100, based on the camera image acquired by the camera image acquisition unit 262. The floor surface setting unit 268 accepts a user operation to edit the height of the floor surface detected as the play area by the play area detection unit 266, and changes the height of the floor surface in accordance with the operation.

[0041] The play area editing unit 270 accepts a user operation to edit the play area automatically detected by the play area detection unit 266, and changes the shape of the play area in accordance with the operation. For example, the play area editing unit 270 reduces the play area automatically detected by the play area detection unit 266 in accordance with the user operation.

[0042] In parallel with the play area detection unit 266 detecting the play area, the map generation unit 274 generates a map for estimating the user's position based on the camera images acquired by the camera image acquisition unit 262. The map generation unit 274 stores the generated map data in the map storage unit 258. The map generation unit 274 terminates map generation when a predetermined number or more of camera images, which are multiple camera images capturing the space around the user from multiple directions, are input.

[0043] The notification unit 278 notifies the App execution unit 290 of information necessary for the App execution unit 290 to execute the VR game. The position estimation unit 280 estimates the position of the user in the real world based on the map data stored in the map storage unit 258 and the camera image acquired by the camera image acquisition unit 262, in other words, estimates the position of the user in the play area.

[0044] For example, the position estimation unit 280 may compare multiple key frames included in the map data with the camera image and estimate the user's position based on the comparison result and the position and line of sight direction of the head mounted display 100 associated with each key frame. Alternatively, the position estimation unit 280 may estimate the user's position using a known self-position estimation technique such as SLAM (Simultaneous Localization and Mapping). The warning processing unit 282 executes a warning process for the user according to the relationship between the boundary of the play area and the user's position.

[0045] Next, the operation of the image display system realized by the above configuration will be described. Fig. 7 is a flowchart showing the operation of the image generation device 200 when setting a play area. The user can select initial setting or resetting of the play area in the system setting menu of the head-mounted display 100. When initial setting or resetting of the play area is selected, the play area setting unit 264 of the image generation device 200 causes the head-mounted display 100 to display a message urging the user to look around via the display control unit 292.

[0046] In response to this, when the user wears the head mounted display 100 on their head and moves while looking around, the head mounted display 100 sequentially transmits data of a plurality of camera images to the image generation device 200. Each of the data of the plurality of camera images includes sensor data at the time of generation of each camera image. The sensor data includes measurements by the motion sensor 134, such as the angular velocity and acceleration of the head mounted display 100. The camera image acquisition unit 262 of the image generation device 200 acquires the camera image data transmitted from the head mounted display 100 (S10).

[0047] The play area detection unit 266 of the image generation device 200 automatically detects a play area in the space surrounding the user based on the camera image acquired in S10 (S12). Specifically, the play area detection unit 266 may estimate the three-dimensional shape of the user's room using a known method based on the camera image and sensor data corresponding to the camera image. Based on the estimated three-dimensional shape of the room, the play area detection unit 266 may detect a plane (typically the floor) perpendicular to the direction of gravity indicated by the sensor data, and detect the result of combining multiple detected planes of the same height as the play area. The play area detection unit 266 stores play area data including coordinate values ​​of a point cloud that constitutes the boundary of the play area in the play area storage unit 256.

[0048] When detecting the play area, the play area detection unit 266 detects the height of the floor surface as the play area. The height of the floor surface may be, for example, the distance between the floor surface and the head mounted display 100 in the direction of gravity. The play area detection unit 266 stores data indicating the height of the floor surface in the play area storage unit 256. When the position of the head mounted display 100 is the origin, the height of the floor surface may be minus 1 meter, for example.

[0049] In parallel with the processing of S12, the map generation unit 274 of the image generation device 200 generates a map for estimating the user's position based on the camera image acquired in S10 (S14). The play area detection unit 266 and the map generation unit 274 repeat the processing of S12 and S14 using new camera images until a predetermined condition indicating that sufficient map data for estimating the user's position has been obtained is met (N in S16). Once sufficient map data has been obtained, the play area detection unit 266 ends the play area detection processing, and the map generation unit 274 ends the map generation processing (Y in S16).

[0050] Next, the floor surface setting unit 268 of the image generating device 200 generates a floor surface adjustment screen based on the data indicating the floor surface height stored in the play area storage unit 256. The floor surface adjustment screen may include an AR image in which an object indicating the floor surface (for example, a translucent grid-like object) is superimposed on a camera image acquired by the camera image acquisition unit 262. The floor surface setting unit 268 displays the floor surface adjustment screen on the display panel of the head-mounted display 100 via the display control unit 292. The floor surface setting unit 268 accepts a user's operation to adjust the floor surface height input on the floor surface adjustment screen, and changes the floor surface height in accordance with the user's operation. The floor surface setting unit 268 stores data indicating the changed floor surface height in the play area storage unit 256 (S18).

[0051] Next, the play area editing unit 270 of the image generating device 200 generates an image showing the play area automatically detected by the play area detection unit 266. Specifically, the play area editing unit 270 generates a play area editing screen based on the play area data stored in the play area memory unit 256. The play area editing screen includes an AR image in which an object showing the play area is superimposed on a camera image acquired by the camera image acquisition unit 262. The display control unit 292 of the image generating device 200 displays the play area editing screen on the display panel of the head-mounted display 100 (S20).

[0052] The play area editing unit 270 accepts a user operation to edit the play area input to the play area editing screen 60 (S22). Specifically, the play area editing unit 270 changes the shape of the play area in accordance with the user operation, that is, enlarges or reduces the play area in accordance with the user operation. The play area editing unit 270 updates the play area data stored in the play area storage unit 256 by storing data of the play area after the shape change (for example, coordinate values ​​of the point cloud that constitutes the boundary) in the play area storage unit 256.

[0053] FIG. 8 schematically illustrates the user's behavior during the loop processing of S10 to S14 in FIG. 7. The user wears the head-mounted display 100 on their head and moves around the room 30 while looking around. The user 32 in FIG. 8 indicates the user's position at this time. Note that the user 32 in FIG. 8 does not represent all of the positions to which the user has moved, but only the positions where key frames, described below, were generated. The arrow 34 indicates the user's line of sight. The play area 36 indicates an example of a play area to be set. While the user is moving as shown in the figure, the play area setting unit 264 of the image generating device 200 displays the camera image acquired by the camera image acquisition unit 262 on the display panel of the head-mounted display 100 via the display control unit 292. This performs video see-through, showing the user the real-world scene in the direction the user is facing. This enhances user safety.

[0054] Fig. 9 shows an example of a camera image acquired by camera image acquisition unit 262 in the process of S10 in Fig. 7. Fig. 10 shows an example of key frame data generated from the camera image of Fig. 9. Map generation unit 274 extracts a plurality of feature points 44 included in camera image 40 using a known method such as a corner detection method, and generates key frame 42 indicating the extracted plurality of feature points 44. When map generation unit 274 extracts 24 or more feature points 44 from one camera image 40, map generation unit 274 stores key frame 42 including the extracted 24 or more feature points 44 in map storage unit 258 in association with the user's position and gaze direction identified by sensor data.

[0055] The map generation unit 274 also identifies the layout and shape of surrounding objects by acquiring the three-dimensional coordinates of feature points extracted from each camera image using well-known techniques such as stereo matching. This three-dimensional spatial information is known as an environmental map. The map generation unit 274 also stores the identified three-dimensional spatial information in the map storage unit 258 as map data.

[0056] The map generator 274 stores multiple criteria (hereinafter referred to as "bins") for deriving a score for the proportion of the user's surrounding space that is covered by the input camera image (in this embodiment, keyframes generated from the camera image). The multiple bins are multiple criteria that divide the user's surrounding space in different ways.

[0057] FIG. 11 shows multiple bins that can be used in this embodiment. Each of bins 50, 52, and 54 has the user's position at its center and divides the user's surrounding space with the direction of gravity as its axis. Bin 50 is a standard that divides the user's surrounding space into four parts with a central angle of 90 degrees based on the user's line of sight at the start of play area setting and a direction perpendicular to the line of sight. Bin 52 is shifted in phase by 45 degrees from bin 50. Bin 54 is a standard that divides the user's surrounding space into 16 parts with a central angle of 22.5 degrees. Hereinafter, the divided areas of the user's surrounding space defined by bins 50, 52, and 54 will be referred to as "divided areas."

[0058] The map generation unit 274 derives a score relating to the proportion of the space around the user that is covered by the input multiple camera images (in the embodiment, keyframes generated from the camera images) based on bins 50, 52, and 54. For example, in S16 of Fig. 7, if the score is equal to or greater than a predetermined threshold, the map generation unit 274 determines that sufficient map data has been obtained and ends the map generation process.

[0059] Specifically, when the map generation unit 274 generates a keyframe from a camera image, it considers that the divided areas that match the line of sight direction in which the keyframe was obtained are filled in for each of bins 50, 52, and 54, and counts the total number of filled divided areas in each bin as the score. When the map generation unit 274 generates a new keyframe, it adds the number of newly filled divided areas in each bin to the score using a similar algorithm.

[0060] For example, when the score is below the threshold "10", the map generation unit 274 determines that the map data is insufficient. This allows the play area detection process and the map generation process to continue. When the score is equal to or greater than the threshold "10", the map generation unit 274 determines that sufficient map data has been acquired and ends the map generation process. This also ends the play area detection process. The score threshold "10" is a value that can be cleared by a user wearing the head-mounted display 100 looking around 180 degrees, but is a value that cannot be cleared if the user does not look around.

[0061] In this manner, until it is determined that sufficient map data has been acquired, the play area detection unit 266 gradually sets the boundaries of the play area in parallel with the map generation unit 274 generating map data. Figure 12 schematically shows how the play area detection unit 266 sets the boundaries of the play area. The play area detection unit 266 basically identifies the presence of an object by detecting a surface that is approximately perpendicular to the floor surface in the map generated by the map generation unit 274.

[0062] In the illustrated example, as the user wearing the head-mounted display 100 looks around, the three-dimensional position coordinates of the feature points of obstacles (e.g., obstacles 70a and 70b) photographed in the line of sight (e.g., directions S and S') are gradually identified. As a result, the area of ​​real space in which the presence of obstacles is identified expands, and the play area detection unit 266 simultaneously sets the play area boundary 72 in front of the detected obstacles, etc. Note that in the illustration, the play area boundary 72 is represented by a line on the floor, but in reality it may have a surface perpendicular to the floor surface.

[0063] Based on the criteria using the bins described above, the play area detection unit 266 updates the play area boundary 72 as new obstacles are detected, as appropriate, until it is determined that sufficient map data has been obtained. As shown in the figure, an additional obstacle 74 may be detected in front of the play area boundary 72 that was previously set, i.e., inside the play area. For example, this may occur when the height of the obstacle 74 is low and an obstacle 70b in the back is detected, but the obstacle 74 does not fall within the field of view of the stereo camera 110.

[0064] In this case, the play area detection unit 266 may change the boundary 72 of the play area to a boundary 76 located in front of the obstacle 74. However, in this case, the play area may become narrower than necessary, which may reduce the enjoyment of the game. Therefore, the play area detection unit 266 modifies the play area according to rules that prevent the play area from becoming narrower than necessary.

[0065] Specifically, the play area detection unit 266 limits the area behind the obstacle 74 as seen from the head-mounted display 100 when the obstacle 74 is detected, and modifies the play area boundary to exclude that area from the play area. In other words, the play area detection unit 266 prevents the areas on the left and right of the obstacle 74 (side areas) as seen from the head-mounted display 100 from being excluded from the play area.

[0066] 13A and 13B are diagrams illustrating an example of a procedure for correcting a play area when the play area detection unit 266 detects another obstacle within the set play area. Similar to FIG. 12, (a) shows a situation in which a new obstacle has been detected in front of the provisional play area boundary 72. Here, contour 80 indicates the contour obtained when the three-dimensional shape of the newly detected obstacle is projected onto the floor. The play area detection unit 266 generates contour 80 based on the three-dimensional position coordinates of the obstacle's feature points.

[0067] Furthermore, the play area detection unit 266 determines the position where a perpendicular line from the center of gravity of the head mounted display 100 when a new obstacle is detected intersects with the floor surface as observation point H. Observation point H corresponds to the position of the user on the floor surface. Next, as shown in (b), the play area detection unit 266 generates tangents 82a and 82b from observation point H to the contour 80, and determines points of contact A and B between the tangents 82a and 82b and the boundary 72, as well as points of intersection C and D between the tangents 82a and 82b and the boundary 72.

[0068] Then, as shown in (c), the play area detection unit 266 deletes the portion of the original boundary 72 between intersections C and D, and sets as a new boundary the thick lines AC and BD and the portion of the contour 80 between tangent points A and B on the side closer to observation point H. As a result, as shown in (d), a new play area boundary 86 is formed, excluding the newly detected obstacle and the area 84 behind it. By modifying the play area using this procedure, it is possible to avoid collisions with the obstacle while allowing the user to move as close to the side of the obstacle as possible.

[0069] Although the figure shows two-dimensional processing on the floor surface, as described above, the boundary of the play area is set in three-dimensional space, including an axis perpendicular to the floor surface. That is, the new play area boundary 86 may actually be composed of a plane perpendicular to the floor surface. In this case, the observation point H is the position of the head-mounted display 100 or the user when an obstacle is detected, and the tangents 82a and 82b to the contour 80 can be considered two vertical planes tangent to the obstacle. Furthermore, the portion of the contour 80 between the tangent points A and B can be considered a third vertical plane that circumscribes the obstacle and intersects with the two vertical planes on the user side. The same applies to the boundaries shown in Figures 14 and 15.

[0070] FIG. 14 shows a schematic diagram of the revised play area when two new obstacles are detected. The play area detection unit 266 independently applies the same procedure as in FIG. 13 to the play area boundary for each of the obstacles 300a and 300b. The new boundary 301 thus formed allows the area between the obstacles 300a and 300b to remain as a play area as much as possible. As a result, collisions with these obstacles can be avoided while minimizing the reduction in the play area. The same applies when three or more obstacles are detected.

[0071] 13, the shape of the boundary after the correction depends on the position of the head mounted display 100 when a new obstacle is detected. Qualitatively, the farther the position of the head mounted display 100 is from the obstacle when the obstacle is detected, the narrower the apparent width of the obstacle, and therefore the narrower the width of the play area to be deleted can also be made.

[0072] For example, in a game in which the user does not move dynamically, it is expected that a certain distance between the head mounted display 100 and the obstacle will be maintained even during gameplay. Therefore, even if the play area around the obstacle is left somewhat wider, it is reasonable to limit the reduction in the play area for obstacles detected far away, in the sense that there is a low possibility that the user will deviate from the play area due to momentum and collide with the obstacle.

[0073] On the other hand, if an obstacle is detected at close range and its apparent width becomes wider, there is a risk that the area behind it will be excluded from the play area more than necessary. Therefore, the play area detection unit 266 checks the distance from the head-mounted display 100 when an obstacle is detected using a predetermined indicator and switches the procedure for correcting the play area boundary. This prevents a situation in which the play area is reduced more than necessary depending on the position of the head-mounted display 100 when an obstacle is detected, and ensures that the play area can be stably secured.

[0074] 15 is a diagram illustrating another example of the procedure for correcting the play area when the play area detection unit 266 detects another obstacle inside the set play area. The diagram is expressed in the same way as in FIG. 13, and (a) shows a situation in which a new obstacle has been detected in front of the provisional play area boundary 72. The contour 302 is the contour when the three-dimensional shape of the newly detected obstacle is projected onto the floor surface, and the observation point H represents the position where the perpendicular line from the center of gravity of the head mounted display 100 intersects with the floor surface when the new obstacle is detected.

[0075] When a new obstacle is detected, the play area detection unit 266 evaluates the distance from the head-mounted display 100 using a predetermined index, and if it is determined that the distance is closer than a predetermined standard, it adopts the correction procedure shown in this figure. In the figure, tangents 304a and 304b from the observation point H to the contour 302 are generated, and the angle θ between one of them and a center line 308 passing through the center point 306 of the internal area of ​​the contour 302 from the observation point H is calculated. o is used as an index of the distance between the head mounted display 100 and the obstacle.

[0076] Naturally, the angle θ o The larger the angle θ, the closer the distance. This index also takes into account the original size of the obstacle, and from a comprehensive perspective, it can be seen as an index that indicates the likelihood that the play area will be reduced more than necessary. However, the index used is not limited to this, and it is also possible to simply use the distance between the observation point H and the contour 302, etc. o When using the index, for example, the angle θo The threshold θ is 45°, etc. th If it is larger, the play area detection unit 266 determines that the distance between the head mounted display 100 and the obstacle is short, and therefore that the play area is likely to be reduced more than necessary using the method of FIG.

[0077] The angle θ o is the threshold θ th A larger upper threshold may also be set, i.e., angle θ o is close to 180°, the observation point H may be in contact with the contour 302 or overlap with the inner region of the contour 302. In this case, the contour 302 is likely not a real obstacle but originates from the user's own body. Therefore, the angle θ o is greater than a predetermined threshold close to 180°, the play area detection unit 266 does not modify the play area boundary. When the distance between the observation point H and the contour 302 is used as an index, the play area detection unit 266 does not modify the play area boundary when the distance is equal to or less than a threshold such as 0.

[0078] Angle θ o is below the upper threshold, and the threshold θ for switching the correction method is set to 45°. th If the angle is larger than the normal angle 2θ, the play area detection unit 266 detects the angle 2θ between the normal and normal tangents 304a and 304b instead of the normal tangents 304a and 304b. o ' is the angle 2θ o The tangent lines 312a and 312b are symmetrical with respect to the center line 308 passing through the observation point H, and the angle between the tangent lines 312a and 312b and the center line 308 is θ o '(<θ o ) are translated so that they are tangent to the contour 302.

[0079] Angle θ ois set to, for example, 25°. The play area detection unit 266 then finds points of contact A' and B' between the new tangents 312a and 312b and the contour 302, and points of intersection C' and D' between the tangents 312a and 312b and the boundary 72. Then, as shown in (c), the play area detection unit 266 deletes the portion of the original boundary 72 between points of intersection C' and D', and sets the straight lines A'C' and B'D' and the portion of the contour 302 between points of contact A' and B' on the side closer to the observation point H as the new boundary.

[0080] As a result, as shown in (d), a new play area boundary 316 is formed, excluding the newly detected obstacle and the area behind it 314. This reduces the amount of reduction in the play area compared to correcting the play area using the original tangent lines 304a and 304b. Note that the illustrated procedure is an example, and the line tangent to the contour 302 may be a polygonal line or a curve, for example. However, the illustrated method makes it possible to reliably remove obstacles from the play area with simple calculations.

[0081] 16 is a flowchart showing the processing steps performed by the play area detection unit 266 to detect the play area. This processing corresponds to S12 in FIG. 7. The play area detection unit 266 determines whether a new obstacle or a new surface of a detected obstacle has been detected based on feature points extracted from the camera image (S30). If no new obstacle or surface has been detected, the play area detection unit 266 ends the processing (N in S30). If a new obstacle or surface has been detected (Y in S30), the play area detection unit 266 determines whether the detected obstacle or surface is in front of the provisionally set play area boundary, i.e., inside the play area (S32).

[0082] In detail, the play area detection unit 266 checks whether the obstacle is located between the head-mounted display 100 when the obstacle was detected in S30 and the boundary of the play area that was provisionally set. If such a positional relationship is not found (N in S32), the play area detection unit 266 determines that the play area boundary in that direction has not yet been set, sets the play area boundary in front of the obstacle, and ends the processing (S36). Although not shown in the figure, if the detected obstacle is located behind the already-set play area boundary, the play area detection unit 266 may end the processing immediately.

[0083] When the detected obstacle or surface is inside the play area (Y in S32), the play area detection unit 266 determines whether the distance between the head mounted display 100 and the obstacle when the obstacle was detected in S30 is equal to or greater than a reference value (S34). o is used as an index, and the threshold θ th The distance is determined to be equal to or greater than the reference value in the following cases: If the distance between the head mounted display 100 and the obstacle is equal to or greater than the reference value (Y in S34), the play area detection unit 266 corrects the play area boundary using the tangent to the outline of the obstacle that passes through the observation point H, as shown in FIG. 13 (S38).

[0084] Specifically, a new boundary is generated using the portion of the obstacle's contour between the tangent points and a part of the tangent line with the tangent point as its endpoint. If the distance between the head mounted display 100 and the obstacle is shorter than the reference value (N in S34), the play area detection unit 266 corrects the play area boundary using the tangent line of the obstacle's contour, which has a narrower angle than the tangent line passing through the observation point H, as shown in Fig. 15 (S40).

[0085] Specifically, the play area detection unit 266 generates two lines symmetrical to the center line of the obstacle's contour, forming a smaller angle than the tangent line of S38, and translates them until they are tangent to the contour. A new boundary is then generated using the portion of the obstacle's contour between the tangent points and a part of the tangent line with the tangent point as its endpoint. By repeating the illustrated process each time a new obstacle is detected, the boundary of the play area can finally be completed. As described above, it is desirable to allow the user to adjust the play area set by the play area detection unit 266.

[0086] FIG. 17 shows an example of a play area editing screen. The play area editing screen 60 includes a play area 62 and a boundary 64. The play area 62 is an image showing the play area (typically a floor surface without obstacles) and may be, for example, an image showing a translucent grid-like object. The boundary 64 is an image showing the boundary of the play area 62 and is an image that intersects the play area perpendicularly at the boundary of the play area 62. The boundary 64 may also be, for example, a translucent grid-like object. As described above, the play area editing unit 270 accepts user operations to transform, enlarge, or reduce the displayed play area, and stores the edited play area data in the play area storage unit 256.

[0087] 18 is a flowchart showing the operation of the image generation device 200 when a VR game is being executed. This flowchart starts when the user wears the head-mounted display 100 and performs an operation to start the VR game on the image generation device 200. In response to this, the notification unit 278 of the image generation device 200 transmits data of the play area stored in the play area storage unit 256, for example, data indicating the shape and size of the play area, to the App execution unit 290 (S50).

[0088] The App execution unit 290 then reads out the program data for the VR game from the App storage unit 254 and starts the VR game (S52). For example, the App execution unit 290 may place a game item within a play area in a virtual space depicting the game world, thereby allowing the user to obtain the item. The display control unit 292 displays an image of the VR game (e.g., a VR image) generated by the App execution unit 290 on the display panel of the head-mounted display 100.

[0089] The camera image acquisition unit 262 of the image generation device 200 sequentially acquires the camera images sequentially transmitted from the head-mounted display 100, and the position estimation unit 280 sequentially estimates the user's position and gaze direction in the real world (S54). The notification unit 278 notifies the app execution unit 290 of the user's position and gaze direction estimated by the position estimation unit 280 (S56). The app execution unit 290 progresses the VR game in accordance with the estimated user's position and gaze direction (S58). For example, the app execution unit 290 may move a user character in a VR game in accordance with changes in the user's position in the real world (in other words, within the play area).

[0090] When the user's position in the real world approaches the boundary of the play area, for example, when the distance from the head mounted display 100 worn by the user to the boundary of the play area becomes equal to or less than a predetermined threshold (for example, 30 cm) (Y in S60), the warning processing unit 282 of the image generating device 200 detects this. At this time, the warning processing unit 282 executes a predetermined warning process for the user (S62).

[0091] For example, when the user's position approaches the boundary of the play area, the warning processing unit 282 may pass an image indicating the boundary of the play area (for example, the boundary 64 shown in FIG. 17 ) to the display control unit 292. The display control unit 292 may display, as a display image, on the display panel of the head-mounted display 100, a game image generated by the app execution unit 290 with the image indicating the boundary of the play area superimposed thereon. Furthermore, when the user's position in the real world approaches the boundary of the play area or passes beyond the boundary of the play area, the warning processing unit 282 may display a video see-through image on the display panel of the head-mounted display 100 via the display control unit 292.

[0092] Furthermore, the warning processing unit 282 may first display an image showing the boundary of the play area when the user's position in the real world approaches the boundary of the play area (for example, 30 centimeters), and then display a video see-through image when the user's position in the real world approaches the boundary of the play area even closer (for example, 10 centimeters). On the other hand, if the user's position in the real world is not near the boundary of the play area (N in S60), S62 is skipped. If a predetermined termination condition is met (Y in S64), such as when the user stops the execution of the VR game, the flow in this figure is terminated. If the termination condition is not met (N in S64), the flow returns to S54 and the VR game continues.

[0093] According to the present embodiment described above, the image generation device sets a play area in which a user wearing a head-mounted display can move around safely. Specifically, the image generation device detects obstacles based on images captured by a stereo camera mounted on the head-mounted display, and sets the boundaries of the play area to exclude the obstacles. If a new obstacle is detected within the play area that has already been set, the image generation device sets a new play area boundary using a tangent line from the head-mounted display to the outline of the obstacle at the time of detection.

[0094] This allows the minimum necessary area, namely the obstacle and the area behind it as seen from the head-mounted display, to be excluded from the play area. Since the position of the head-mounted display at the time of detection is estimated to be close to the user's position when playing a game or other content, by leaving the area in front of the obstacle in the play area and excluding the area behind it, it is possible to avoid collisions with obstacles while minimizing the reduction in the actual play area.

[0095] Furthermore, if the distance between the head-mounted display and an obstacle at the time of detection is determined to be short, the angle between the tangents is narrowed to prevent the expansion of the area excluded from the play area. This minimizes the effect that the position of the head-mounted display at the time of obstacle detection has on the size of the play area, making it easy to set a play area that is sufficient for any situation.

[0096] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0097] 100 Head-mounted display, 200 Image generation device, 256 Play area memory unit, 264 Play area setting unit, 266 Play area detection unit, 270 Play area editing unit, 274 Map generation unit, 280 Position estimation unit, 282 Warning processing unit, 292 Display control unit.

Claims

1. a camera image acquisition unit that acquires data of a camera image showing a surrounding space of the user captured by a camera mounted on the head mounted display; a play area setting unit that detects obstacles in the surrounding space based on the camera image and sets a play area in which a user can move based on the result of the detection; a warning processing unit that executes a warning process for a user according to a relationship between a position of the user playing an application using the head mounted display and a boundary of the play area; Equipped with The play area setting unit determines the positional relationship between the boundary of the set play area and a newly detected obstacle, and decides whether to modify the boundary of the play area based on the result, and when the obstacle is inside the play area, limits the obstacle and the area behind it as seen from the head-mounted display at the time of detection, and modifies the boundary so as to exclude it from the play area.

2. The information processing device described in claim 1, characterized in that the play area setting unit defines the boundary after modification as a line on the floor surface consisting of two tangent lines that pass through the position of the head-mounted display at the time of detection and are tangent to the contour of the obstacle, and the part of the contour between the tangent lines on the side closer to the head-mounted display.

3. The information processing device according to claim 1 or 2, characterized in that the play area setting unit checks the distance between the obstacle and the head-mounted display at the time of detection using a predetermined indicator, and switches the method of correcting the boundary depending on the result.

4. The information processing device described in claim 2, characterized in that the play area setting unit checks the distance between the obstacle and the head-mounted display at the time of detection using a predetermined indicator, and when it determines that the distance is closer than a standard, determines the modified boundary using two other tangents that form an angle smaller than the angle formed by the two tangents passing through the position of the head-mounted display.

5. The information processing device described in claim 4, characterized in that the play area setting unit generates the other two tangents by translating two lines that pass through the position of the head-mounted display at the time of detection and form a predetermined angle symmetrical with the center line of the contour so as to be tangent to the contour.

6. The information processing device according to any one of claims 3 to 5, wherein the play area setting unit determines the closeness of the distance using the magnitude of the angle formed by the two tangents passing through the position of the head mounted display as an index.

7. An information processing device according to any one of claims 1 to 6, characterized in that when there are multiple obstacles inside the play area, the play area setting unit independently limits each obstacle and the area behind it and modifies the boundary to exclude them from the play area.

8. A step of acquiring data of a camera image showing a surrounding space of the user, the camera image being captured by a camera mounted on the head mounted display; Detecting obstacles in the surrounding space based on the camera image, and setting a play area in which a user can move based on the result of the detection; executing a warning process for a user according to a relationship between a position of the user playing an application using the head mounted display and a boundary of the play area; Including, An information processing method characterized in that the step of setting the play area includes determining a positional relationship between the boundary of the set play area and a newly detected obstacle, and deciding whether to modify the boundary of the play area based on the result, and when the obstacle is inside the play area, limiting the obstacle and the area behind it as seen from the head-mounted display at the time of detection, and modifying the boundary so as to exclude it from the play area.

9. A function for acquiring data of camera images of the user's surroundings taken by a camera mounted on the head mounted display; and A function of detecting obstacles in the surrounding space based on the camera image and setting a play area in which a user can move based on the result of the detection; a function of executing a warning process for a user according to a relationship between a position of the user playing an application using the head mounted display and a boundary of the play area; This is realized on a computer. A computer program characterized in that the play area setting function determines the positional relationship between the boundary of the set play area and a newly detected obstacle, and decides whether to modify the boundary of the play area based on the result, and when the obstacle is inside the play area, limits the obstacle and the area behind it as seen from the head-mounted display at the time of detection, and modifies the boundary so as to exclude it from the play area.

Images