Information processing device and information processing method

Abstract

To easily and appropriately set a play area in which a user wearing a head mounted display can move.SOLUTION: A play area detection unit of an image generation device detects a play area 420 on the basis of an image captured by a stereo camera of a head mounted display and causes a user to confirm it. A play area editing unit receives an editing operation of drawing a new boundary line 424 of the play area and detects an edited play area 426. If a size of its circumscribed rectangle 428 exceeds an upper limit, the play area editing unit crops the play area 426 along a cutting line 430 vertical to an adjustment axis x to determine a final play area 432.SELECTED DRAWING: Figure 12

Description

【Technical Field】 【0001】 The present invention relates to an information processing apparatus and an information processing method for processing data related to a head-mounted display. 【Background Art】 【0002】 An image display system in which a user wearing a head-mounted display can view a target space from a free viewpoint has become widespread. For example, there is known electronic content that realizes virtual reality (VR) by using a virtual three-dimensional space as a display target and displaying an image corresponding to the user's line-of-sight direction on the head-mounted display. By using a head-mounted display, it is also possible to enhance the sense of immersion in video or improve the operability of applications such as games. In addition, a walk-through system has been developed in which a user wearing a head-mounted display can virtually walk around in a space displayed as a video by physically moving. 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0003】 The range within which a user wearing a head-mounted display can move during play of an application needs to be restricted according to the size of the user's room and the environment such as furniture and luggage placed in the user's room. However, it is difficult to set a play area that can achieve both safety and entertainment while taking into account such individual circumstances of the user. [[ID=2,4]] 【0004】 [[ID=II6]] 【Means for Solving the Problems】 【0005】 ​To solve the above problems, one aspect of the present invention relates to an information processing device. This information processing device is characterized by comprising: a storage unit that stores data of a play area that a user can move in while playing an application in the space surrounding the user wearing a head-mounted display; a display control unit that displays an image of the play area stored in the storage unit on the head-mounted display; and a play area editing unit that receives user operations to edit the play area, detects when the upper limit of the play area's size is exceeded, and adjusts the size by cropping a portion of it. 【0006】 Another aspect of the present invention relates to an information processing method. This information processing method is characterized by including the steps of: reading data from memory of a play area that a user can move in while playing an application in the space surrounding a user wearing a head-mounted display; displaying an image representing the play area on the head-mounted display; and receiving user input to edit the play area, detecting that the upper limit of the play area's size has been exceeded, and adjusting the size by cropping a portion of it. 【0007】 Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between systems, computer programs, recording media on which computer programs are recorded in a readable manner, data structures, etc., are also valid embodiments of the present invention. [Effects of the Invention] 【0008】 According to the present invention, a play area that a user wearing a head-mounted display can move around in can be easily and appropriately configured. [Brief explanation of the drawing] 【0009】 [Figure 1] This figure shows an example of the appearance of the head-mounted display in this embodiment. [Figure 2] This figure shows an example of the configuration of the image display system in this embodiment. [Figure 3]This figure illustrates an example of the image world that the image generation device displays on the head-mounted display in this embodiment. [Figure 4] This figure shows the internal circuit configuration of the image generation device in this embodiment. [Figure 5] This figure shows the internal circuit configuration of the head-mounted display in this embodiment. [Figure 6] This block diagram shows the functional blocks of the image generation device in this embodiment. [Figure 7] This flowchart shows the operation of the image generation device when setting the play area in this embodiment. [Figure 8] This figure illustrates the play area detection method used by the play area detection unit of this embodiment. [Figure 9] This figure shows an image of the play area displayed in S20 of Figure 7. [Figure 10] This diagram schematically illustrates how a user edits a play area in this embodiment. [Figure 11] This figure illustrates the movement of the controller cursor and the changes in the play area in this embodiment. [Figure 12] This figure illustrates a method by which the play area editing unit adjusts the size of the play area after it has been edited by the user in this embodiment. [Figure 13] This figure illustrates an example of an image that the play area editing unit displays on a head-mounted display during the editing period of the play area in this embodiment. [Figure 14] This diagram illustrates how, in this embodiment, the play area editing department monitors for exceeding the play area size while the user is drawing the boundaries of the play area. [Figure 15] This diagram illustrates how the Play Area Editorial Department, in this embodiment, cuts out areas of a new play area other than those added by the user. [Figure 16]This diagram illustrates the policy for determining the cutting lines when a user edits the play area from outside the play area in this embodiment. [Figure 17] This figure illustrates how the Play Area Editorial Department determines the cutting lines based on the shape of the play area and the actual objects present in it, according to this embodiment. [Figure 18] This flowchart shows the processing procedure in which the play area editing unit updates the play area data in response to editing operations by the user in this embodiment. [Modes for carrying out the invention] 【0010】 This embodiment relates to an image display system that displays application images on a head-mounted display worn on the user's head. Figure 1 shows an example of the appearance of the head-mounted display 100. The head-mounted display 100 in this embodiment consists of an output mechanism 102 and a mounting mechanism 104. The mounting mechanism 104 includes a mounting band 106 that wraps around the user's head when worn, securing the device in place. 【0011】 The output mechanism 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and has a display panel inside that faces the eyes when worn. The display panel of the head-mounted display 100 in this embodiment is opaque. In other words, the head-mounted display 100 is a light-opaque type head-mounted display. 【0012】 Inside the housing 108, there may further be an eyepiece lens that is positioned between the display panel and the user's eyes when the head-mounted display 100 is worn and that expands the user's viewing angle. The head-mounted display 100 may further include a speaker or earphone at a position corresponding to the user's ears when worn. Also, the head-mounted display 100 incorporates a motion sensor and detects the translational and rotational movements of the head of the user wearing the head-mounted display 100, and thus the position and orientation at each moment. 【0013】 Also, the head-mounted display 100 includes a stereo camera 110 on the front surface of the housing 108. The stereo camera 110 captures a moving image of the surrounding real space within a field of view corresponding to the user's line of sight. If the captured image is immediately displayed, so-called video see-through can be realized, where the user can see the state of the real space in the direction they are facing as it is. Furthermore, if a virtual object is drawn on the image of a real object shown in the captured image, augmented reality (AR) can be realized. 【0014】 Figure 2 shows a configuration example of the image display system according to the present embodiment. The image display system includes a head-mounted display 100, an image generation device 200, and a controller 140. The head-mounted display 100 is connected to the image generation device 200 by wireless communication. The image generation device 200 may further be connected to a server via a network. In that case, the server may provide data of an online application such as a game that multiple users can participate in via the network to the image generation device 200. 【0015】 The image generation device 200 is an information processing device that identifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, generates a display image so as to have a corresponding field of view, and outputs it to the head-mounted display 100. For example, the image generation device 200 may generate a virtual world, which is the stage of the game, as a display image while advancing an electronic game, or may display a moving image for viewing or information provision regardless of whether it is a virtual world or the real world. Also, by displaying a panoramic image with a wide viewing angle centered on the user's viewpoint on the head-mounted display 100, a deep sense of immersion in the displayed world can be given to the user. Note that the image generation device 200 may be a stationary game machine or a PC. 【0016】 The controller 140 is a controller (for example, a game controller) that is held by the user's hand and to which user operations for controlling image generation in the image generation device 200 and image display in the head-mounted display 100 are input. The controller 140 is connected to the image generation device 200 by wireless communication. As a modification, one or both of the head-mounted display 100 and the controller 140 may be connected to the image generation device 200 by wired communication via a signal cable or the like. 【0017】 FIG. 3 is a diagram for explaining an example of an image world that the image generation device 200 displays on the head-mounted display 100. In this example, a state where the user 12 is in a room that is a virtual space is created. In the world coordinate system that defines the virtual space, as shown in the drawing, objects such as walls, floors, windows, tables, and objects on the table are arranged. The image generation device 200 defines a view screen 14 in the world coordinate system according to the position of the viewpoint and the direction of the line of sight of the user 12, and draws a display image by representing an image of the object there. 【0018】 The image generation device 200 acquires the position and direction of the user 12's viewpoint (hereinafter, these may be collectively referred to as "viewpoint") from the head-mounted display 100 at a predetermined rate, and changes the position and direction of the view screen 14 accordingly. This allows the image to be displayed on the head-mounted display 100 within the field of view corresponding to the user's viewpoint. Furthermore, the image generation device 200 can generate a stereo image with parallax and display the stereo image in the left and right areas of the display panel of the head-mounted display 100, allowing the user 12 to experience the virtual space in 3D. This allows the user 12 to experience virtual reality as if they were actually in a room in the displayed world. 【0019】 Figure 4 shows the internal circuit configuration of the image generation device 200. The image generation device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and main memory 226. These components are interconnected via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228. 【0020】 The communication unit 232 includes peripheral device interfaces such as USB and IEEE1394, and network interfaces such as wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive and non-volatile memory. The output unit 236 outputs data to the head-mounted display 100. The input unit 238 receives data input from the head-mounted display 100 and also receives data input from the controller 140. The recording medium drive unit 240 drives removable recording media such as magnetic disks, optical disks, or semiconductor memory. 【0021】 The CPU 222 controls the entire image generation device 200 by executing the operating system stored in the memory unit 234. The CPU 222 also executes various programs (e.g., VR game applications) read from the memory unit 234 or a removable recording medium and loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has both geometry engine and rendering processor functions, performing drawing processing according to drawing commands from the CPU 222 and outputting the drawing results to the output unit 236. The main memory 226 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing. 【0022】 Figure 5 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU 120, main memory 122, display unit 124, and audio output unit 126. These units are interconnected via a bus 128. An input / output interface 130 is further connected to the bus 128. The input / output interface 130 is connected to a communication unit 132, which includes a wireless communication interface, a motion sensor 134, and a stereo camera 110. 【0023】 The CPU 120 processes information acquired from various parts of the head-mounted display 100 via the bus 128, and also supplies display images and audio data acquired from the image generation device 200 to the display unit 124 and the audio output unit 126. The main memory 122 stores the programs and data necessary for processing by the CPU 120. 【0024】 The display unit 124 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the head-mounted display 100. The display unit 124 may achieve stereoscopic vision by displaying a pair of stereo images in areas corresponding to the left and right eyes. The display unit 124 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, and which expand the user's field of view. 【0025】 The audio output unit 126 consists of speakers or earphones positioned to correspond to the user's ears when the head-mounted display 100 is worn, allowing the user to hear audio. The communication unit 132 is an interface for sending and receiving data with the image generation device 200, and communication is achieved using known wireless communication technologies such as Bluetooth®. The motion sensor 134 includes a gyro sensor and an accelerometer, and acquires the angular velocity and acceleration of the head-mounted display 100. 【0026】 As shown in Figure 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from the left and right viewpoints, with a field of view corresponding to the user's viewpoint. The images captured by the stereo camera 110, which show the space around the user, are hereinafter also referred to as "camera images." Camera images can also be described as images that show objects located in the direction of the user's line of sight (typically in front of the user). The measured values ​​from the motion sensor 134 and the data of the images captured by the stereo camera 110 (camera images) are transmitted to the image generation device 200 via the communication unit 132 as needed. 【0027】 In this embodiment of the image display system, a play area is defined that specifies the range of the real world in which a user wearing a head-mounted display 100 can move while playing an application. The play area can be described as the area or range within the user's surrounding space (the real world space surrounding the user) in which the user is permitted to move around while viewing virtual reality images (hereinafter also referred to as "VR images"). If the user attempts to deviate from the play area or has deviated from it while playing an application, the image display system provides the user with a warning that serves as an alert or prompts them to return to the play area. 【0028】 The above application is a game application that displays VR images on a head-mounted display 100, and will hereinafter also be referred to as a "VR game". For example, the VR game may be a tennis game that displays a VR image of a virtual tennis court and changes the position of the character on the virtual tennis court in accordance with the user's movement (walking, etc.) in the real world. 【0029】 In this embodiment, the image generation device 200 automatically detects the play area. At this time, the image generation device 200 determines the range in which the user can move without colliding with objects by identifying the positions and shapes of surrounding objects in real space captured by the stereo camera 110 of the head-mounted display 100. Furthermore, the image generation device 200 accepts user input to edit the automatically detected play area and changes the shape of the play area according to that input. This allows the user to efficiently set a play area of ​​any shape. 【0030】 Figure 6 is a block diagram showing the functional blocks of the image generation device. As described above, the image generation device 200 performs general information processing such as managing the progress of the VR game and communicating with the server, but Figure 6 shows in detail the functional blocks related to setting the play area. At least some of the functions of the image generation device 200 shown in Figure 6 may be implemented on a server connected to the image generation device 200 via a network, or on the head-mounted display 100. 【0031】 Furthermore, the multiple functional blocks shown in Figure 6 can be implemented in hardware using the configuration shown in Figure 4, such as the CPU 222, GPU 224, main memory 226, and storage unit 234, and in software using a computer program that implements the functions of the multiple functional blocks. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof, and are not limited to any one of these. 【0032】 The image generation device 200 comprises a data processing unit 250 and a data storage unit 252. The data processing unit 250 performs various data processing operations. The data processing unit 250 transmits and receives data to and from the head-mounted display 100 and the controller 140 via the communication unit 232, output unit 236, and input unit 238 shown in Figure 4. The data storage unit 252 stores data that is referenced or updated by the data processing unit 250. 【0033】 The data storage unit 252 includes an App storage unit 254, a play area storage unit 256, and a map storage unit 258. The App storage unit 254 stores data for an application that generates VR images (in this embodiment, a VR game). The play area storage unit 256 stores data related to the play area. The data related to the play area includes data indicating the positions of the point cloud that constitutes the boundary of the play area (for example, the coordinate values ​​of each point in the world coordinate system). 【0034】 The map storage unit 258 stores map data for estimating the position of the head-mounted display 100 (i.e., the position of the user wearing the head-mounted display 100). The map data in this embodiment is an image (feature points shown in the image) showing the room in which the user plays the VR game, and includes a set of images (feature points shown in the image) whose positional relationships (position and direction) are known. Specifically, the map data includes multiple sets of pairs that associate the position of the head-mounted display 100, the direction of gaze, and keyframes. The map data may also include other items necessary for so-called self-localization. 【0035】 A set of keyframes is data that represents the features of the image seen at the same position and viewing direction. A keyframe is an image generated from camera images captured by the stereo camera 110 of the head-mounted display 100, and is an image that contains a predetermined number or more feature points. The number of feature points that should be included in a keyframe may be 24 or more. Feature points may include corners detected by known corner detection methods, or they may be detected based on the brightness gradient. A keyframe can also be described as a collection of partial images extracted from a camera image. 【0036】 The data processing unit 250 comprises a system unit 260, an application execution unit 290, and a display control unit 292. The functions of these multiple functional blocks may be implemented in a computer program. The CPU 222 and GPU 224 of the image generation device 200 may perform the functions of the multiple functional blocks by reading the computer program from the storage unit 234 or recording medium into the main memory 226 and executing it. 【0037】 The App execution unit 290 reads data for an application selected by the user (a VR game in this embodiment) from the App storage unit 254 and executes the application selected by the user. The display control unit 292 transmits various image data (e.g., VR images and AR images) generated by the system unit 260 and the App execution unit 290 to the head-mounted display 100 and displays these images on the display unit 124 (display panel) of the head-mounted display 100. The display control unit 292 may also transmit audio data to the head-mounted display 100 and output it from the audio output unit 126 of the head-mounted display 100. 【0038】 The system unit 260 performs system processing related to the head-mounted display 100. The system unit 260 provides common services to multiple applications for the head-mounted display 100 (e.g., VR games). The system unit 260 includes a camera image acquisition unit 262, a play area setting unit 264, a map generation unit 274, a notification unit 278, a position estimation unit 280, and a warning processing unit 282. 【0039】 The camera image acquisition unit 262 acquires camera images captured by the stereo camera 110 of the head-mounted display 100, which are transmitted from the head-mounted display 100. The play area setting unit 264 performs various processes related to setting the play area. Specifically, the play area setting unit 264 sets the play area based on the camera images acquired by the camera image acquisition unit 262 and the user's operations input via the controller 140. The play area setting unit 264 includes a play area detection unit 266, a floor setting unit 268, and a play area editing unit 270. 【0040】 The play area detection unit 266 automatically detects the play area from the space surrounding the user wearing the head-mounted display 100, based on the camera image acquired by the camera image acquisition unit 262. The floor setting unit 268 accepts user input to edit the height of the floor surface detected as a play area by the play area detection unit 266, and changes the height of the floor surface according to that input. 【0041】 The play area editing unit 270 receives user input to edit the play area automatically detected by the play area detection unit 266, and changes the shape of the play area according to that input. For example, the play area editing unit 270 shrinks or expands the play area automatically detected by the play area detection unit 266 according to user input. 【0042】 The map generation unit 274 generates a map for estimating the user's position based on camera images acquired by the camera image acquisition unit 262, in parallel with the play area detection unit 266 detecting the play area. The map generation unit 274 stores the generated map data in the map storage unit 258. The map generation unit 274 terminates map generation when it receives a predetermined number of camera images, which are multiple camera images capturing the user's surroundings from multiple directions. 【0043】 The notification unit 278 notifies the App execution unit 290 of the information necessary for the App execution unit 290 to execute the VR game. The position estimation unit 280 estimates the user's position in the real world, or in other words, the user's position in the play area, based on the map data stored in the map storage unit 258 and the camera image acquired by the camera image acquisition unit 262. 【0044】 For example, the position estimation unit 280 may compare multiple keyframes included in the map data with camera images and estimate the user's position based on the comparison result and the position and gaze direction of the head-mounted display 100 associated with each keyframe. Alternatively, the position estimation unit 280 may estimate the user's position using known self-localization techniques such as SLAM (Simultaneous Localization and Mapping). The warning processing unit 282 executes a warning process for the user according to the relationship between the play area boundary and the user's position. 【0045】 Next, the operation of the image display system realized by the above configuration will be explained. Figure 7 is a flowchart showing the operation of the image generation device 200 when setting the play area. The user can select to initialize or reset the play area in the system settings menu of the head-mounted display 100. If initial initialization or resetting of the play area is selected, the play area setting unit 264 of the image generation device 200 causes the head-mounted display 100 to display a message prompting the user to look around via the display control unit 292. 【0046】 In response, when the user wears the head-mounted display 100 on their head and moves around while looking around, the head-mounted display 100 sequentially transmits data from multiple camera images to the image generation device 200. Each of the data from multiple camera images includes sensor data generated at the time of each camera image generation. The sensor data includes measurements from the motion sensor 134, such as the angular velocity and acceleration of the head-mounted display 100. The camera image acquisition unit 262 of the image generation device 200 acquires the camera image data transmitted from the head-mounted display 100 (S10). 【0047】 The play area detection unit 266 of the image generation device 200 automatically detects the play area in the space surrounding the user based on the camera image acquired in S10 (S12). Specifically, the play area detection unit 266 may estimate the three-dimensional shape of the user's room using a known method based on the camera image and the sensor data corresponding to the camera image. Based on the estimated three-dimensional shape of the room, the play area detection unit 266 may detect a plane (typically the floor) perpendicular to the direction of gravity indicated by the sensor data, and detect the result of combining multiple detected planes of the same height as the play area. The play area detection unit 266 stores the play area data, including the coordinate values ​​of the point cloud constituting the boundary of the play area, in the play area storage unit 256. 【0048】 The play area detection unit 266 detects the height of the floor surface as the play area when a play area is detected. The height of the floor surface may be, for example, the distance between the floor surface and the head-mounted display 100 in the direction of gravity. The play area detection unit 266 stores data indicating the height of the floor surface in the play area storage unit 256. If the position of the head-mounted display 100 is the origin, the height of the floor surface may be minus 1 meter, for example. 【0049】 The map generation unit 274 of the image generation device 200 generates a map for estimating the user's position based on the camera image acquired in S10, in parallel with the processing in S12 (S14). The play area detection unit 266 and the map generation unit 274 repeat the processing in S12 and S14 using the new camera image until a predetermined condition is met indicating that sufficient map data for estimating the user's position has been obtained (N in S16). Once sufficient map data has been obtained, the play area detection unit 266 terminates the play area detection process, and the map generation unit 274 terminates the map generation process (Y in S16). 【0050】 Next, the floor setting unit 268 of the image generation device 200 generates a floor adjustment screen based on data indicating the height of the floor stored in the play area storage unit 256. The floor adjustment screen may include an AR image in which an object representing the floor (for example, a semi-transparent grid-like object) is superimposed on a camera image acquired by the camera image acquisition unit 262. The floor setting unit 268 displays the floor adjustment screen on the display panel of the head-mounted display 100 via the display control unit 292. The floor setting unit 268 accepts user input to adjust the height of the floor entered on the floor adjustment screen and changes the height of the floor according to the user's input. The floor setting unit 268 stores data indicating the changed floor height in the play area storage unit 256 (S18). 【0051】 Next, the play area editing unit 270 of the image generation device 200 generates an image showing the play area automatically detected by the play area detection unit 266. Specifically, the play area editing unit 270 generates a play area editing screen based on the play area data stored in the play area storage unit 256. The play area editing screen includes an AR image in which an object showing the play area is superimposed on a camera image acquired by the camera image acquisition unit 262. The display control unit 292 of the image generation device 200 displays the play area editing screen on the display panel of the head-mounted display 100 (S20). 【0052】 The play area editing unit 270 accepts user input for editing the play area entered on the play area editing screen (S22). Specifically, the play area editing unit 270 changes the shape of the play area in response to user input, that is, expands or shrinks the play area in response to user input. The play area editing unit 270 updates the play area data stored in the play area storage unit 256 by storing the data of the play area after the shape change (for example, the coordinate values ​​of the point cloud that constitutes the boundary) in the play area storage unit 256. 【0053】 Figure 8 is a diagram illustrating the play area detection method used by the play area detection unit 266. The play area detection unit 266 basically identifies the presence of an object by detecting a surface that is approximately perpendicular to the floor surface in the map generated by the map generation unit 274. In the example shown in the figure, as the user wearing the head-mounted display 100 looks around, the three-dimensional position coordinates of the feature points of obstacles (e.g., obstacles 70a, 70b) captured in their line of sight direction (e.g., directions S, S') are gradually identified. 【0054】 As a result, the area of ​​real space where obstacles are found expands, and the play area detection unit 266 sets the play area boundary 72 in front of the detected obstacles. Until it is determined that sufficient map data has been obtained in S16 of Figure 7, the play area detection unit 266 updates the play area boundary 72 as needed in response to the detection of new obstacles. Once it is determined that sufficient map data has been obtained, such as after completing a full circuit, the play area detection unit 266 designates the play area at that point as the provisional play area. 【0055】 Figure 9 shows an image of the play area displayed in S20 of Figure 7. The play area image 60 includes a play area portion 62 and a boundary portion 64. The play area portion 62 is an image showing the play area (typically an unobstructed floor surface), and may, for example, be an image showing a semi-transparent grid-like object. The boundary portion 64 is an image showing the boundary surface of the play area, and represents an image of a surface that intersects the play area perpendicularly at the boundary of the play area. The boundary portion 64 may also, for example, be a semi-transparent grid-like object. 【0056】 Figure 10 schematically shows how a user edits a play area. The play area editing unit 270 displays an AR image on the head-mounted display 100 by superimposing objects representing the play area, such as the play area unit 62 shown in Figure 9, onto the camera image. In other words, the user 400 sees the play area object 404, which has been provisionally determined by the play area detection unit 266, on the floor surface surrounded by real objects (for example, real object 402). 【0057】 The user 400 uses the controller 140 to arbitrarily modify the boundaries of the original play area by drawing a new virtual line 406 on the floor. To this end, the play area editor 270, for example, emits a virtual beam 408 from the controller 140 and displays an AR image representing the object of the controller cursor 410 on the floor where it lands. As the user moves the controller 140 like a laser pointer, the play area editor 270 moves the controller cursor 410 and represents the movement trajectory as the line 406. This gives the user the sensation of drawing a line on the floor. 【0058】 Figure 11 illustrates the movement of the controller cursor and the changes in the play area. The figure shows an overview of the play area 62 at each stage of the editing period. As shown in (a) to (c), the user moves the controller cursor 410 on the floor and draws a line indicating its movement trajectory, for example, a line that goes out of the play area 62 and then back in. In this case, as shown in (d), the play area editing unit 270 adds the area enclosed by the drawn line to the play area 62. In this case, the user is drawing a new boundary line of the play area by moving the controller cursor 410 on the floor. 【0059】 In response, the play area editing unit 270 generates new play area data including the added area and updates the original data stored in the play area storage unit 256. When shrinking the play area, the play area editing unit 270 accepts an operation to draw a line that divides the original play area section 62. In this case, the play area editing unit 270 will, for example, designate the side of the divided play area section 62 where the user is located as the new play area. 【0060】 The play area editing unit 270 accepts editing operations for the play area, as illustrated, not only when the play area detection unit 266 automatically detects a play area, but also when a play area that has been finalized through editing operations is modified at a later time. This increases the degree of flexibility in setting up the play area. 【0061】 On the other hand, if the play area can be infinitely expanded in a vast space without surrounding obstacles, the following problems may arise. Specifically, if the image generation device 200 and the head-mounted display 100 are connected by a cable, a long cable will be required, which will hinder the user's movement. In some cases, there is a risk that the cable may get tangled in the body or cause the user to trip. Even if the image generation device 200 and the head-mounted display 100 are connected wirelessly, the communication status may become unstable depending on the user's position. 【0062】 Furthermore, tracking user movements can become computationally intensive and tracking accuracy can deteriorate, leading to display delays and degradation of content processing accuracy. Therefore, in this embodiment, an upper limit is set on the size of the play area. The upper limit of the play area is, for example, a rectangle of 5 x 5 m. If a user sets a play area that exceeds the upper limit through editing operations, the play area editing unit 270 detects this and adjusts the size to fit within the upper limit by cropping the edited play area. This eliminates the need to restart the editing operation from the beginning. 【0063】 Figure 12 illustrates how the play area editing unit 270 adjusts the size of the play area after it has been edited by the user. The figure shows an overview of the play area (or its object) at each stage of the editing process. (a) shows the initial state, where the bounding rectangle 422 of the play area 420 is within the upper limit on both sides. (b) shows the state during user editing, where a new boundary line 424 is drawn on the original play area 420 by moving the controller cursor. 【0064】 (c) shows the controller cursor inside the original play area 420. As a result, the play area editor 270 recognizes a new play area 426 that includes the area inside the drawn boundary line. The play area editor 270 may also recognize the timing when the user starts and stops pressing the trigger button on the controller 140 as the start and end of boundary line drawing. In other words, the play area editor 270 may enable boundary line drawing when the trigger button is pressed. 【0065】 (c) When the user finishes drawing, the play area editing unit 270 virtually generates a bounding rectangle 428 around the new play area 426 and determines whether its size exceeds the upper limit. If the size of the bounding rectangle 428 exceeds the upper limit, that is, if at least one of the side lengths exceeds the upper limit, the play area editing unit 270 sets a cutting line 430 to cut out the excess portion and cuts out the new play area 426. 【0066】 Here, the cutting line 430 is a line perpendicular to the sides of the circumscribed rectangle 428 that exceed the upper limit (two opposing sides). In the following explanation, the axis x parallel to the sides of the circumscribed rectangle 428 that exceed the upper limit will be called the "adjustment axis". In the diagram, the adjustment axis is shown as being in only one direction, but if both adjacent sides of the circumscribed rectangle 428 exceed the upper limit, there will be two adjustment axes perpendicular to each other. (d) shows the play area 432 after being cut by the cutting line 430, and by setting the cutting line 430 appropriately, the circumscribed rectangle 434 can be kept within the upper limit. 【0067】 By setting the cutting line 430 such that the length of the side in the direction of the adjustment axis of the circumscribing rectangle 434 is limited to the upper limit xu, the maximum play area can be provided in the direction the user wishes to expand. The size of the play area can be checked using a circumscribing polygon, circle, ellipse, etc., in addition to the circumscribing rectangle. The cutting line can also be in any direction with respect to the adjustment axis, and the cutting line can be a polyline or a curve. However, with the settings shown in the figure, the play area can be adjusted immediately with simple calculations. 【0068】 Figure 13 illustrates the images that the play area editing unit 270 displays on the head-mounted display 100 during the play area editing period. (a) is an example of an image immediately after the user has drawn the boundaries of a new play area using the controller 140, and the play area object 440 is represented. The play area object 440 is actually superimposed on the image of the corresponding floor area in the camera image. 【0069】 For example, the user, while viewing the camera image and controller cursor displayed on the head-mounted display 100, uses the controller 140 to move the controller cursor from the front left to the back, and then to the front right. The play area editor 270 recognizes this movement trajectory as a new boundary of the play area, and by temporarily filling the inside of that boundary with play area objects, an object 440 as shown in the diagram is displayed. 【0070】 When the Play Area Editor 270 recognizes that the user has finished drawing, such as by releasing the trigger button on the controller 140, it checks the size of the bounding rectangle, as shown in Figure 12(c). If it exceeds the upper limit, the Play Area Editor 270 sets a cutting line perpendicular to the adjustment axis and cuts out the play area. In the figure, for illustrative purposes, the set cutting line 442 is shown aligned with the play area object 440, but it does not actually need to be displayed. 【0071】 The play area editor 270 then represents the remaining play area and its boundary surface, which are cut outside the cutting line 442, as objects for play area section 444 and boundary section 446, as shown in (b). The play area section 444 and boundary section 446 are also actually represented by being superimposed on the camera image. Through the image changes shown, the user can recognize that the boundary line they drew exceeded the upper limit size of the play area. 【0072】 On the other hand, the play area editor 270 may more clearly indicate that the play area has been cut out. For example, the play area editor 270 may make the color of the part 448 of the play area 444 that overlaps with the cut line, or the boundary part 446 above it, different from the rest. Alternatively, the play area editor 270 may display an animation showing how a part of the play area is cut out during the transition from (a) to (b), or in image (b), the cut-out part may be left as an object with higher transparency than the play area 444. 【0073】 The play area editing unit 270 may also provide the user with a predetermined warning sound via the display control unit 292 and the head-mounted display 100 at the time the play area is cropped. In any case, these indications allow the user to easily recognize that the size has been adjusted because the initially drawn boundary line exceeded the upper limit. Alternatively, the play area editing unit 270 may constantly monitor for exceeding the play area size while the user is drawing the boundary line and notify the user when it exceeds the limit. 【0074】 Figure 14 illustrates how the play area editing unit 270 monitors for exceeding the play area size while the user is drawing the play area boundary. Assume the user is using the controller 140 to draw a new boundary 452 on the original play area 450. At this time, the play area editing unit 270 generates a bounding rectangle 454 on the original play area 450 and boundary 452 at a predetermined rate. In other words, if the boundary 452 extends, the bounding rectangle 454 may also be updated accordingly. 【0075】 The play area editing unit 270 then notifies the user when the size of the bounding rectangle 454 exceeds the upper limit, that is, when at least one of the side lengths exceeds the upper limit. For example, the play area editing unit 270 may make the user hear a predetermined warning sound via the display control unit 292 and the head-mounted display 100. Alternatively, the play area editing unit 270 may make the border of the part that exceeds the upper limit a different color from the others, or it may vibrate the controller 140. These notifications allow the user to easily take action, such as changing the direction of the border being drawn. 【0076】 In the example in Figure 12, the area to be cropped was exclusively the area added by the user through drawing boundaries. As shown in Figure 13, this allows the user to intuitively understand that a portion of the play area has been cropped in the direction they are primarily looking. On the other hand, if the original play area is close to its maximum size, there may be no room to expand the play area in the direction the user wants to add. In this case, if only the area added by the user is cropped, the entire added area may be cropped, resulting in a situation where the play area remains unchanged. Taking such cases into consideration, the Play Area Editor 270 does not need to limit the area to be cropped to only the area added by the user. 【0077】 Figure 15 illustrates how the play area editing unit 270 cuts out the area of ​​the new play area other than the area added by the user. (a) shows the same state as in Figure 12, where the user has drawn a boundary line on the original play area 420 and the play area editing unit 270 has generated a bounding rectangle 428 on the new play area 426. The user is assumed to be at position 460 within the play area. In this example, the play area editing unit 270 sets a cut line 464 with respect to the adjustment axis x so as to cut out the play area on the opposite side from where the user added the area. 【0078】 This approach allows the play area to move in the direction the user wants to add, even when the play area is close to its maximum size, making it easier to reflect the user's intentions. On the other hand, in this case, the area being cut off is more likely to be outside the user's field of view. As a result, the user may find themselves at the edge of the play area or even outside of it without realizing it. 【0079】 Therefore, the play area editing unit 270 may determine the cutting lines so that the distance between the user's position 460 and the boundary line of the play area remains above a predetermined value, as shown in (b). That is, the play area editing unit 270 sets the cutting lines on the adjustment axis x at positions where the distances x1 and x2 between the user's position 460 and the cutting lines are above a predetermined value. In the example shown in the figure, two cutting lines 466a and 466b are set on both sides of the play area with respect to the adjustment axis x. 【0080】 Note that the lower limits given to intervals x1 and x2 may differ on the side where the user added an area and the opposite side. Also, as shown in (a), the side where the user did not add an area may be prioritized for trimming, and if the interval x1 between the trimming line and the user falls below the lower limit, the remaining portion to be trimmed may be trimmed on the side where the user added an area. Alternatively, the priority may be reversed. In this way, it is guaranteed that the user remains within the play area regardless of which area is prioritized for trimming with respect to the adjustment axis. 【0081】 Figure 16 illustrates the policy for determining the cut line when a user edits a play area from outside the play area. Similar to Figure 12, the user draws a boundary line around the original play area 420, and the play area editing unit 270 generates a bounding rectangle 428 around the new play area 426. However, in this example, the user is initially located at position 470 outside the play area 420. In this case, the play area editing unit 270 determines the cut line 472 to cut off the play area on the side furthest from the user's position 470 with respect to the adjustment axis x. 【0082】 In other words, the Play Area Editorial Department 270 ensures that the distance D from the user to the play area does not change through cropping. This prevents a situation where the entire play area moves further away from the user as the user tries to expand the play area's range in a direction away from them. 【0083】 Figure 17 is a diagram illustrating how the play area editing unit 270 determines the cutting lines based on the shape of the play area and the actual objects present. Both (a) and (b) show, similar to Figure 12, the state in which the user draws a boundary line on the original play area 420 and the play area editing unit 270 generates a bounding rectangle 428 for the new play area 426. In (a), the play area editing unit 270 determines the cutting lines 480a and 480b so as to prioritize cutting out portions of the new play area 426 that are smaller than a predetermined width. 【0084】 The play area editor 270 scans the boundary lines of the bounding rectangle 428 in the vertical and horizontal directions, for example, and detects locations where the distance between opposing boundary lines (e.g., distances d1 and d2) is less than a threshold. It then prioritizes determining a cut line 480b that excludes such locations. This eliminates play areas that the user cannot pass through and that are essentially meaningless. For this purpose, the threshold given to the distance between opposing boundary lines is approximately the width of a person's shoulders, for example, 30-40 cm. 【0085】 In (b), the play area editing unit 270 determines the cutting lines 484a and 484b so as to prioritize cutting out the portion containing the physical object 482 within the new play area 426. In this case, the play area editing unit 270 refers to the map stored in the map memory unit 258 and detects the presence of any physical object 482 that is even partially inside the new play area 426. This ensures that areas with a risk of collision are reliably eliminated, and a safe play area can be set up. 【0086】 Note that in both (a) and (b), the area that should be prioritized for cutting out is located in the area added by the user. However, if there is a similar area on the opposite side of the adjustment axis x, that area may also be used for cutting. Furthermore, if cases (a) and (b) occur simultaneously, both may be used for cutting. The cutting line should be determined in at least one of these cases, and if necessary, the cutting line may be further determined using one of the policies shown in Figures 12, 15, and 16. In the example shown in the figure, assuming the user is within the play area, cutting lines 480a and 484a are also provided on the opposite side of the adjustment axis to preserve as much of the play area as possible in the direction the user wants to expand. 【0087】 Figure 18 is a flowchart showing the processing procedure by which the play area editing unit 270 updates the play area data in response to editing operations by the user. This process corresponds to S20 and S22 in the flowchart shown in Figure 7. First, the play area editing unit 270 represents the play area detected by the play area detection unit 266 as an object as shown in Figure 9, and displays it on the head-mounted display 100 by superimposing it on the camera image (S30). 【0088】 When the user uses the controller 140 to draw a line on the floor, the play area editing unit 270 detects this and draws an object indicating a new boundary line at the corresponding location (S32). For example, if the user changes the direction indicated by the controller 140 while holding down the trigger button, the play area editing unit 270 moves the controller cursor object in accordance with the change and extends its movement trajectory. After the controller cursor leaves the original play area and returns to the original play area, and the trigger button is released, the play area editing unit 270 recognizes the movement trajectory at that point as a new boundary line. 【0089】 The play area editor 270 then determines whether the size of the new play area, including the area added by the new boundary line, exceeds the upper limit (S34). Specifically, the play area editor 270 generates a bounding rectangle around the new play area and determines whether the length of its sides exceeds the upper limit. If the length of at least one side exceeds the upper limit (Y in S34), the play area editor 270 determines the cutting line of the play area. 【0090】 Specifically, as shown in Figure 17, the play area editing unit 270 checks whether there are areas that should be preferentially cut out based on the shape of the play area and the presence of real objects within the play area (S36). If such areas exist (Y in S36), the play area editing unit 270 sets a cutting line that excludes those areas (S38). For example, the play area editing unit 270 sets a cutting line at a predetermined distance on the adjustment axis from areas where the width is smaller than a threshold or where real objects exist. 【0091】 If there are physical objects within the play area, the play area editor 270 may set a cut line to exclude the relevant area before confirming the size of the play area in S34. After setting the cut line in S38, or if there are no areas that should be cut preferentially (N in S36), the play area editor 270 sets the cut lines in a predetermined order of priority (S40). Possible options for the order of priority include prioritizing cutting from areas newly added by the user, as shown in Figure 12; prioritizing cutting from the opposite side, as shown in Figure 15(a); or prioritizing cutting from areas farther away from the user when the user is outside the play area, as shown in Figure 16. 【0092】 When the user is within the play area, preferably, a constraint is set such that the distance between the cutting line on the adjustment axis and the user is greater than or equal to a predetermined value, as shown in Figure 15(b). If, by setting the cutting line in S38 to exclude the area that should be cut first, it is expected that the size of the new playalia will fit within the upper limit, the process in S40 can be omitted. 【0093】 The play area editor 270 cuts out the play area using the cut lines set in S38 and S40, and updates the data of the original play area (S42). If the size of the new play area does not exceed the upper limit in S34 (N in S34), the data is also updated with the new play area (S42). The play area editor 270 then represents the newly determined play area as an object and displays it on the head-mounted display 100 by superimposing it on the camera image (S44). 【0094】 In S38 and S40, if a cutting line is set and the play area generated by user operation is cut, the play area editing unit 270 may display an image to indicate the cut area, as shown in Figure 13(b). Also, in S32, even when the user is drawing the boundary line, the user may be notified with an image or sound when the size exceeds the upper limit. 【0095】 According to the embodiment described above, the image processing device automatically detects a play area in which a user wearing a head-mounted display can move around safely, and accepts editing operations from the user. The image processing device checks whether the size of the newly generated play area is within a predetermined upper limit, and if it exceeds the upper limit, it automatically sets an appropriate cutting line and cuts it to bring the size of the play area to an appropriate value. This prevents situations in which the user fails to grasp the size of an effective play area, resulting in editing failures and forcing them to perform editing work multiple times. 【0096】 Furthermore, the image processing device crops the play area at an optimal position according to the direction in which the user wants to add a region and the user's position. This allows the device to determine a play area that is less prone to errors caused by editing operations while still capturing the user's intentions. Therefore, even if the user edits the play area relatively roughly and freely, there is a high probability that it will ultimately be accepted as a proper play area, thus lowering the psychological barrier to editing operations. 【0097】 The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of symbols] 【0098】 100 Head-mounted display, 200 Image generation device, 256 Play area storage unit, 264 Play area setting unit, 266 Play area detection unit, 268 Floor surface setting unit, 270 Play area editing unit, 274 Map generation unit, 280 Position estimation unit, 282 Warning processing unit, 292 Display control unit.

Claims

[Claim 1] A storage unit that stores data on the play area that a user can move in while playing an application in the space surrounding the user wearing a head-mounted display, A display control unit that causes the head-mounted display to display an image of the play area stored in the memory unit, A play area editing unit accepts user input to edit the aforementioned play area, detects when the play area exceeds its size limit, and adjusts the size by cropping a portion of it. An information processing device characterized by comprising: [Claim 2] The information processing device according to claim 1, characterized in that when the length of the side of the rectangle circumscribing the edited play area exceeds the upper limit, the play area editing unit cuts out the edited play area with a cutting line perpendicular to the side. [Claim 3] The information processing device according to claim 2, characterized in that when the user is in the edited play area, the play area editing unit sets the cutting line at a position where the distance from the user is greater than or equal to a predetermined value on an adjustment axis parallel to the side exceeding the upper limit. [Claim 4] The information processing device according to claim 3, characterized in that the aforementioned play area editorial department sets the cutting lines at two locations on the adjustment axis. [Claim 5] The information processing apparatus according to any one of claims 1 to 4, characterized in that the display control unit displays the image showing the cropped play area on the head-mounted display, clearly indicating the cropped portion. [Claim 6] The information processing apparatus according to claim 5, characterized in that the display control unit causes the head-mounted display to show an animation in which a part of the play area is being cut off. [Claim 7] When the user edits the play area, the display control unit superimposes an object representing a cursor that can be moved by the user and its movement trajectory onto the image of the real space captured by the camera of the head-mounted display and displays it on the head-mounted display. The information processing device according to claim 1, characterized in that the play area editing unit accepts the movement trajectory as a new boundary line of the play area, and when the length of the sides of the rectangle circumscribing the play area before editing and the boundary line exceeds an upper limit, it notifies the user of that fact. [Claim 8] The information processing device according to any one of claims 1 to 4, characterized in that the play area editing unit determines an area to be preferentially cut out from the edited play area based on at least one of the shape of the edited play area and the presence of real objects within the play area. [Claim 9] The information processing device according to claim 2, wherein when the user is outside the edited play area, the play area editing unit cuts out the area of ​​the edited play area on the side furthest from the user, along an adjustment axis parallel to the side exceeding the upper limit. [Claim 10] The process involves reading data from memory of the play area that a user can move around in while playing an application in the space surrounding a user wearing a head-mounted display, and The steps include: displaying an image showing the play area on the head-mounted display; The process includes receiving user input to edit the play area, detecting that the play area exceeds its size limit, and adjusting its size by cropping a portion of it. An information processing method characterized by including [Claim 11] A function that reads data from memory of the play area that the user can move around in while playing an application in the space surrounding the user wearing a head-mounted display, A function to display an image indicating the play area on the head-mounted display, The system accepts user input to edit the aforementioned play area, detects when the play area exceeds its size limit, and adjusts its size by cropping a portion of it. A computer program characterized by enabling a computer to implement the following.

Images