Information processing device and information processing method

Abstract

To accurately process contents with image display using a head-mounted display even with a simple device configuration.SOLUTION: An image generation device sequentially obtains frames 150 of a moving image captured by a stereo camera of a head-mounted display, and generates an image 154 for display to be used in video see-through etc. A cropping unit of the image generation device crops a region 152b in which a floor surface appears, and makes an image 156 for setting a play area in which a user can move. The cropping unit also crops a region 152c of a prescribed size in which a front of the head-mounted display appears, and makes an image 158 for obtaining state information pertaining to the head-mounted display.SELECTED DRAWING: Figure 8

Description

Technical Field

[0004] , , , , , ,

[0005] ,

[0001] The present invention relates to an information processing apparatus and an information processing method for realizing image expression by a head-mounted display.

Background Art

[0002] An image display system in which a user wearing a head-mounted display can view a target space from a free viewpoint has become widespread. For example, there is known electronic content that realizes virtual reality (VR: Virtual Reality) by using a virtual three-dimensional space as a display target and displaying an image corresponding to the user's line-of-sight direction on the head-mounted display. By using a head-mounted display, it is also possible to enhance the sense of immersion in video and improve the operability of applications such as games. In addition, a walk-through system has been developed in which a user wearing a head-mounted display can virtually walk around in a space displayed as a video by physically moving.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In order to realize a high-quality user experience using a head-mounted display, it is required to constantly and accurately grasp the state of real objects, such as the position and posture of the user, and the positional relationship with furniture and walls in the surroundings. On the other hand, the more information that needs to be acquired, and the more the accuracy of the information is improved, the more necessary equipment such as sensors increases, which causes problems in terms of manufacturing cost, weight, power consumption, etc.

[0004] The present invention has been made in view of such problems, and an object thereof is to provide a technique that can accurately process content accompanied by image display by a head-mounted display with a simple device configuration.

Means for Solving the Problems

[0005] To solve the above problems, one aspect of the present invention relates to an information processing device. This information processing device is characterized by comprising: a display control unit that displays a moving image being captured by a camera mounted on a head-mounted display as a display image on the head-mounted display; a cropping unit that cuts out a portion of the area from the frame of the moving image according to a predetermined rule; and an image analysis unit that analyzes the image of the portion of the image and acquires data necessary to display an image different from the moving image on the head-mounted display.

[0006] Another aspect of the present invention relates to an information processing method. This information processing method is characterized by including the steps of: displaying a moving image being captured by a camera mounted on a head-mounted display as a display image on the head-mounted display; extracting a portion of the region from the frame of the moving image according to a predetermined rule; and analyzing the image of the portion of the region to obtain data necessary for displaying an image different from the moving image on the head-mounted display.

[0007] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between systems, computer programs, recording media on which computer programs are recorded in a readable manner, data structures, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0008] According to the present invention, content involving image display via a head-mounted display can be processed accurately even with a simple device configuration. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the appearance of the head-mounted display in this embodiment. [Figure 2] This figure shows an example of the configuration of the image display system in this embodiment. [Figure 3] This figure illustrates an example of the image world that the image generation device displays on the head-mounted display in this embodiment. [Figure 4] This is a diagram illustrating the principles of Visual SLAM. [Figure 5] This figure shows the internal circuit configuration of the image generation device in this embodiment. [Figure 6] This figure shows the internal circuit configuration of the head-mounted display in this embodiment. [Figure 7] This block diagram shows the functional blocks of the image generation device in this embodiment. [Figure 8] This figure illustrates the image changes caused by the cropping in this embodiment. [Figure 9] This figure illustrates a more detailed example of the area to be cropped in this embodiment. [Figure 10] This flowchart shows the procedure by which the image generation device performs various processing using the captured image in this embodiment. [Figure 11] Figure 10 shows a flowchart illustrating the process by which the play area control unit sets the play area in step S20. [Figure 12] This diagram schematically illustrates the user's actions when setting up a play area in this embodiment. [Figure 13] This figure illustrates the image of a frame captured by a stereo camera and the keyframe data obtained therefrom in this embodiment. [Figure 14] This figure illustrates several bins that can be used in this embodiment. [Figure 15] This figure shows an example of the play area editing screen in this embodiment. [Modes for carrying out the invention]

[0010] This embodiment relates to an image display system that displays images on a head-mounted display worn on the user's head. Figure 1 shows an example of the appearance of the head-mounted display 100. In this example, the head-mounted display 100 consists of an output mechanism 102 and a mounting mechanism 104. The mounting mechanism 104 includes a mounting band 106 that wraps around the user's head when worn, securing the device in place.

[0011] The output mechanism 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and has a display panel inside that faces the eyes when worn. In this embodiment, the display panel of the head-mounted display 100 is opaque. That is, the head-mounted display 100 is a light-opaque type head-mounted display.

[0012] The housing 108 may further include an eyepiece positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, to expand the user's field of view. The head-mounted display 100 may also include speakers or earphones positioned to correspond to the user's ears when worn. Furthermore, the head-mounted display 100 incorporates motion sensors to detect the translational and rotational movements of the user's head, as well as its position and orientation at each moment in time.

[0013] Furthermore, the head-mounted display 100 is equipped with a stereo camera 110 on the front of the housing 108. The stereo camera 110 captures video of the surrounding real space with a field of view corresponding to the user's gaze. By displaying the captured images immediately, it is possible to achieve so-called video see-through, where the user can see exactly what is happening in the real space in the direction they are facing. Moreover, by drawing virtual objects on top of the images of real objects captured in the images, augmented reality (AR) can be realized.

[0014] FIG. 2 shows a configuration example of the image display system according to the present embodiment. The image display system 10 includes a head-mounted display 100, an image generation device 200, and a controller 140. The head-mounted display 100 is connected to the image generation device 200 by wireless communication. The image generation device 200 may be further connected to a server via a network. In that case, the server may provide data of an online application such as a game in which a plurality of users can participate via the network to the image generation device 200.

[0015] The image generation device 200 is an information processing device that specifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, and generates a display image so as to have a corresponding visual field and outputs it to the head-mounted display 100. For example, the image generation device 200 may generate a virtual world, which is the stage of the game, as a display image while advancing an electronic game, or may display a moving image for viewing or information provision regardless of whether it is a virtual world or the real world. Further, by displaying a panoramic image with a wide viewing angle centered on the user's viewpoint on the head-mounted display 100, a deep sense of immersion in the display world can be given to the user. Note that the image generation device 200 may be a stationary game machine or a PC.

[0016] The controller 140 is a controller (for example, a game controller) that is held by the user's hand and to which a user operation for controlling image generation in the image generation device 200 and image display in the head-mounted display 100 is input. The controller 140 is connected to the image generation device 200 by wireless communication. As a modification, one or both of the head-mounted display 100 and the controller 140 may be connected to the image generation device 200 by wired communication via a signal cable or the like.

[0017] FIG. 3 is a diagram for explaining an example of an image world that the image generation device 200 causes the head-mounted display 100 to display. In this example, the user 12 is creating a state of being in a room that is a virtual space. In the world coordinate system that defines the virtual space, objects such as walls, floors, windows, tables, and objects on the table are arranged as shown in the figure. The image generation device 200 defines the view screen 14 in the world coordinate system according to the position of the user 12's viewpoint and the direction of the line of sight, and draws a display image by representing the images of the objects thereon.

[0018] The image generation device 200 acquires the state of the head-mounted display 100 at a predetermined rate, and changes the position and orientation of the view screen 14 accordingly. Thereby, an image can be displayed on the head-mounted display 100 in the visual field corresponding to the user's viewpoint. Further, if the image generation device 200 generates a stereoscopic image having parallax and displays the stereoscopic image in the left and right regions of the display panel of the head-mounted display 100, the virtual space can also be stereoscopically viewed by the user 12. Thereby, the user 12 can experience virtual reality as if being in the room of the display world.

[0019] In order to realize the image representation as shown in FIG. 3, it is required to track the changes in the position and orientation of the user's head, and thus the head-mounted display 100, and to accurately control the position and orientation of the view screen 14 accordingly. In addition, for a user wearing a light-impermeable type head-mounted display, since the surrounding situation in the real space cannot be seen, it is necessary to provide means for avoiding dangers such as collisions and stumbles. In the present embodiment, the captured image by the stereo camera 110 used for realizing the above-described video see-through and AR is utilized to efficiently and with low latency acquire the necessary information related to the real object.

[0020] Images used for displays such as video see-throughs are preferably captured with a wide field of view that covers the range of a person's field of vision. In this case, much of the information about real objects around the user, as well as information such as the position and orientation of the user's head relative to those objects, will be included in the captured image. This embodiment utilizes this to extract the necessary parts from the captured image according to the purpose and use them for image analysis, thereby efficiently obtaining the necessary information without the need to provide a separate dedicated sensor. In the following description, at least one of the position and orientation of the head-mounted display 100 may be collectively referred to as the "state" of the head-mounted display.

[0021] Visual SLAM (Simultaneous Localization and Mapping) is a known technique for simultaneously estimating the self-position of a mobile object equipped with a camera and creating an environmental map using captured images. Figure 4 is a diagram illustrating the principle of Visual SLAM. Camera 22 is mounted on the mobile object and captures video of the real space 26 within its field of view while changing its position and orientation. Here, suppose feature points 28a and 28b representing points 24 on the same subject are extracted from frame 20a captured at a certain time and frame 20b captured at time Δt.

[0022] The shift in the positional coordinates of the corresponding feature points 28a and 28b (hereinafter sometimes referred to as "corresponding points") in each frame plane depends on the change in the position and orientation of the camera 22 at time Δt. Specifically, if R and T are matrices representing the changes due to the rotational and translational motions of the camera 22, respectively, and P1 and P2 are the three-dimensional vectors from the camera 22 to point 24 at each time step, then the following relationship holds. P1 = R·P2 + T

[0023] By utilizing this relationship and extracting multiple corresponding points in two frames that are sequentially preceding or succeeding in time, and solving a system of equations, the changes in the position and orientation of camera 22 during that time can be identified. Furthermore, by minimizing the error in the derivation result through recursive calculation, the 3D information of the subject surface in real space 26, such as point 24, can be accurately constructed. If camera 22 is a stereo camera 110, the 3D position coordinates of point 24, etc., can be determined independently at each time point, making calculations such as extracting corresponding points easier.

[0024] However, even if camera 22 is a monocular camera, the Visual SLAM algorithm is well-established. Therefore, for the purpose of tracking the state of the head-mounted display 100, the camera equipped on the head-mounted display 100 is not limited to a stereo camera 110. Furthermore, numerous Visual SLAM algorithms have been proposed, and any of them can be adopted. In any case, according to the principle illustrated, the state change of camera 22 from the previous time can be derived at the same rate as the frame rate of the video.

[0025] On the other hand, such minute movements result in minute changes in the image. Therefore, if a sufficient number of feature points are obtained in each frame, it is sufficient to track the state of the real object without analyzing the entire frame. As a result, by extracting an appropriate region for analysis, it is possible to achieve both tracking accuracy and processing speed.

[0026] Therefore, in this embodiment, as an example, a portion of the image captured by the stereo camera 110 is used to acquire the state of the head-mounted display 100 and to set the play area, which defines the range of the real world in which the user can move, for use in setting the view screen 14. Here, the play area is the range of the real world in which the user, who is viewing the virtual world through the head-mounted display 100, can move around, and is, for example, a range in which safe movement is guaranteed without colliding with surrounding objects.

[0027] Figure 5 shows the internal circuit configuration of the image generation device 200. The image generation device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and main memory 226. These components are interconnected via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228.

[0028] The communication unit 232 includes peripheral device interfaces such as USB and IEEE1394, and network interfaces such as wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive and non-volatile memory. The output unit 236 outputs data to the head-mounted display 100. The input unit 238 receives data input from the head-mounted display 100 and also receives data input from the controller 140. The recording medium drive unit 240 drives removable recording media such as magnetic disks, optical disks, or semiconductor memory.

[0029] The CPU 222 controls the entire image generation device 200 by executing the operating system stored in the memory unit 234. The CPU 222 also executes various programs (e.g., VR game applications) read from the memory unit 234 or a removable recording medium and loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has both geometry engine and rendering processor functions, performing drawing processing according to drawing commands from the CPU 222 and outputting the drawing results to the output unit 236. The main memory 226 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing.

[0030] Figure 6 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU 120, main memory 122, display unit 124, and audio output unit 126. These units are interconnected via a bus 128. An input / output interface 130 is further connected to the bus 128. The input / output interface 130 is connected to a communication unit 132, which includes a wireless communication interface, a motion sensor 134, and a stereo camera 110.

[0031] The CPU 120 processes information acquired from various parts of the head-mounted display 100 via the bus 128, and also supplies display images and audio data acquired from the image generation device 200 to the display unit 124 and the audio output unit 126. The main memory 122 stores the programs and data necessary for processing by the CPU 120.

[0032] The display unit 124 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the head-mounted display 100. The display unit 124 may achieve stereoscopic vision by displaying a pair of stereo images in areas corresponding to the left and right eyes. The display unit 124 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, and which expand the user's field of view.

[0033] The audio output unit 126 consists of speakers or earphones positioned to correspond to the user's ears when the head-mounted display 100 is worn, allowing the user to hear audio. The communication unit 132 is an interface for sending and receiving data with the image generation device 200, and communication is achieved using known wireless communication technologies such as Bluetooth®. The motion sensor 134 includes a gyro sensor and an accelerometer, and acquires the angular velocity and acceleration of the head-mounted display 100.

[0034] As shown in Figure 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints, with a field of view corresponding to the user's viewpoint. The frames of the moving images captured by the stereo camera 110 show objects that are in the direction of the user's line of sight (typically in front of the user). The values ​​measured by the motion sensor 134 and the data of the images captured by the stereo camera 110 are transmitted to the image generation device 200 via the communication unit 132 as needed.

[0035] In the image display system 10 of this embodiment, as described above, a play area is set that defines the range of the real world in which a user wearing the head-mounted display 100 can move while playing an application. If the user attempts to deviate from the play area or has deviated from it while playing an application, the image display system 10 provides the user with a warning that alerts them or prompts them to return to the play area.

[0036] Figure 7 is a block diagram showing the functional blocks of the image generation device. As described above, the image generation device 200 performs general information processing such as managing the progress of electronic games and communicating with a server, but Figure 7 shows in detail the functional blocks related to processing images captured by the stereo camera 110. At least some of the functions of the image generation device 200 shown in Figure 6 may be implemented in the head-mounted display 100, or in a server connected to the image generation device 200 via a network. In addition, the function of the image generation device 200 to extract (crop) frames from captured images and obtain predetermined information may be implemented separately as an information processing device.

[0037] Furthermore, the multiple functional blocks shown in Figure 7 can be implemented in hardware using the configuration shown in Figure 5, such as the CPU 222, GPU 224, main memory 226, and storage unit 234, and in software using a computer program that implements the functions of the multiple functional blocks. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof, and are not limited to any one of these.

[0038] The image generation device 200 includes a data processing unit 250 and a data storage unit 252. The data processing unit 250 performs various data processing operations. The data processing unit 250 transmits and receives data to and from the head-mounted display 100 and the controller 140 via the communication unit 232, output unit 236, and input unit 238 shown in Figure 5. The data storage unit 252 is implemented in the form of a storage unit 234 shown in Figure 5 and stores data that is referenced or updated by the data processing unit 250.

[0039] The data storage unit 252 includes an App storage unit 254, a Play Area storage unit 256, and a Map storage unit 258. The App storage unit 254 stores data such as programs and object models necessary for running applications that involve image display, such as VR games. The Play Area storage unit 256 stores data related to the play area. The data related to the play area includes data indicating the positions of the point cloud that constitutes the boundary of the play area (for example, the coordinate values ​​of each point in the world coordinate system).

[0040] The map storage unit 258 stores registration information for acquiring the position and orientation of the head-mounted display 100, and by extension, the head of the user wearing the head-mounted display 100. Specifically, the map storage unit 258 stores keyframe data used in Visual SLAM and data of an environment map (hereinafter referred to as "map") that represents the structure of an object surface in three-dimensional real space, in association with each other.

[0041] Keyframes are data from frames selected in Visual SLAM based on predetermined criteria, such as having a certain number of feature points or more. However, as mentioned above, a "frame" is a portion of the entire area of ​​the video image captured by the stereo camera 110 that has been cropped according to a predetermined rule. In the following explanation, this portion may also be referred to as a "frame." By using keyframes as "past frames" to compare them with the feature points of the current frame (latest frame), errors that accumulate over time in tracking the position and orientation of the head-mounted display 100 can be canceled out.

[0042] The map data is information about the three-dimensional position coordinates of a group of points representing the surfaces of objects that make up the room in which the user plays the electronic game, and each point is associated with a feature point extracted from a keyframe. The keyframe data is associated with the state of the stereo camera 110 when it was acquired. The number of feature points to be included in the keyframe may be 24 or more. The feature points may include corners detected by known corner detection methods, or they may be detected based on the brightness gradient.

[0043] The data processing unit 250 comprises a system unit 260, an application execution unit 290, and a display control unit 292. The functions of these multiple functional blocks may be implemented in a computer program. The CPU 222 and GPU 224 of the image generation device 200 may perform the functions of the multiple functional blocks by reading the computer program from the storage unit 234 or recording medium into the main memory 226 and executing it.

[0044] The system unit 260 performs system processing related to the head-mounted display 100. The system unit 260 provides common services to multiple applications for the head-mounted display 100 (e.g., VR games). The system unit 260 includes an image acquisition unit 262, an input information acquisition unit 263, a cropping unit 274, a state information acquisition unit 276, and a play area control unit 264.

[0045] The image acquisition unit 262 sequentially acquires frame data of images captured by the stereo camera 110, transmitted from the head-mounted display 100. This data is basically wide-angle image data that can be used for display. The input information acquisition unit 263 acquires the content of user operations via the controller 140. The cropping unit 274 crops the area necessary for subsequent processing from the frames acquired by the image acquisition unit 262.

[0046] The cropping unit 274 internally holds a crop rule storage unit 278. The crop rule storage unit 278 stores crop rules that associate the intended use of the cropped image with the rules for determining the area to be cropped. The cropping unit 274 crops each frame according to the crop rules corresponding to the respective processes, in response to requests from the state information acquisition unit 276 and the play area control unit 264.

[0047] The cropped image from the crop unit 274 may be used not only for acquiring state information of the head-mounted display 100 or setting the play area, but also for other image analysis such as image recognition, or for generating display images. Furthermore, functions that perform image analysis using the cropped image, such as the state information acquisition unit 276 and a part of the play area control unit 264, can be collectively referred to as the "image analysis unit." The image analysis unit here has the characteristic of acquiring data necessary to display images other than video see-through on the head-mounted display 100 using the image cropped by the crop unit 274.

[0048] The state information acquisition unit 276 sequentially acquires state information of the head-mounted display 100 using the Visual SLAM method described above. Specifically, the state information acquisition unit 276 acquires state information of the head-mounted display 100, i.e., position and orientation information, at each time point based on the data of each cropped frame provided by the cropping unit 274 and the data stored in the map storage unit 258. The state information acquisition unit 276 may also integrate the information obtained from image analysis with the measured values ​​from the motion sensor 134 built into the head-mounted display 100 to obtain state information.

[0049] The status information of the head-mounted display 100 is used for setting the view screen during application execution, monitoring the user's approach to the play area boundary, and issuing warnings when the user approaches the play area. Therefore, the status information acquisition unit 276 appropriately supplies the acquired status information to the play area control unit 264, the App execution unit 290, and the display control unit 292 according to the situation at that time.

[0050] The play area control unit 264 sets a real-world area where the user can move safely as the play area, and during the application execution phase, it issues warnings as appropriate when the user approaches its boundary. In setting the play area, the play area control unit 264 generates map data using Visual SLAM or the like, using the data of each frame after cropping provided by the crop unit 274.

[0051] For example, the play area control unit 264 instructs the user wearing the head-mounted display 100 to look around. As a result, real objects such as furniture, walls, and floors around the user are captured by the stereo camera 110. The play area control unit 264 sequentially acquires data from cropped areas from each frame of the captured images and generates map data based on this data.

[0052] The play area control unit 264 also refers to the map data and automatically determines the play area as a range of floor space that does not collide with furniture, walls, etc. The play area control unit 264 may accept user editing operations for the play area by displaying an image representing the boundaries of the determined play area on the head-mounted display 100. In this case, the play area control unit 264 acquires the content of the user operation via the controller 140 via the input information acquisition unit 263 and changes the shape of the play area accordingly.

[0053] The play area control unit 264 ultimately stores the data related to the determined play area in the play area storage unit 256. The play area control unit 264 also associates the generated map data with the keyframe data obtained along with it and stores it in the map storage unit 258, so that the state information acquisition unit 276 can read it at any time later.

[0054] The App execution unit 290 reads and executes application data, such as a VR game selected by the user, from the App storage unit 254. During this process, the App execution unit 290 sequentially acquires state information of the head-mounted display 100 from the state information acquisition unit 276, sets the view screen at the corresponding position and orientation, and renders the VR image. This allows the virtual world of the display target to be represented with a field of view that corresponds to the user's head movements.

[0055] The App execution unit 290 may also generate AR images depending on the application selected by the user. In this case, the App execution unit 290 superimposes and draws virtual objects onto the frame of the captured image acquired by the captured image acquisition unit 262, or onto the frame cropped by the cropping unit 274 in accordance with the display processing. Here, the App execution unit 290 determines the drawing position of the virtual objects based on the state information acquired by the state information acquisition unit 276. This allows the virtual objects to be represented appropriately so as to correspond to the subjects captured in the image.

[0056] The display control unit 292 sequentially transmits data from various images generated by the App execution unit 290, such as VR images and AR image frames, to the head-mounted display 100. The display control unit 292 also transmits to the head-mounted display 100, as needed, images instructing the user to look around when setting the play area, images showing the appearance of the tentatively selected play area and accepting edits, and images warning of approaching the boundary of the play area.

[0057] For example, when setting a play area, the display control unit 292, in accordance with a request from the play area control unit 264, transmits the frame of the captured image acquired by the captured image acquisition unit 262, or the frame data cropped by the cropping unit 274 in accordance with the display processing, to the head-mounted display 100 for display. This enables video see-through, which shows the real space in the direction the user is facing, thereby enhancing user safety. The opportunities for realizing video see-through are not limited to this, and may include periods when the user is outside the play area, before or after the application starts, or when the user requests video see-through, among other things.

[0058] Figure 8 illustrates the image change due to the cropping section 274. As described above, the stereo camera 110 captures the real space in the direction the user is facing with a wide-angle horizontal field of view of approximately 100° or more. Frame 150 represents one frame from a pair of stereo moving images obtained in this way, and shows the room where the user is located.

[0059] When video see-through is achieved, the image acquisition unit 262 may use the frame 150 acquired by the image acquisition unit 262 as the display image 154 without cropping by the cropping unit 274. Alternatively, as shown in the figure, the cropping unit 274 may crop the area 152a excluding a peripheral area of ​​a predetermined width to create the display image 154. In any case, the display control unit 292 applies appropriate corrections to the frame 150 or the display image 154 and transmits it to the head-mounted display 100 as the final display image data.

[0060] For example, as shown in Figure 1, if the stereo camera 110 is located on the lower front of the housing 108 of the head-mounted display 100, and the direction of the optical axis does not coincide with the user's line of sight, displaying it as is may cause discomfort to the user. For this reason, the display control unit 292 performs a conversion process on the frame 150 or the display image 154 so that the field of view corresponds to the user's field of view. In addition, the display control unit 292 may apply corrections that introduce distortion inverse to the distortion and chromatic aberration of the eyepiece so that an image without distortion or color shift is visible when viewed through the eyepiece of the head-mounted display 100.

[0061] On the other hand, when setting a play area, the cropping unit 274 crops the area 152b that shows the floor surface and provides it to the play area control unit 264 as an image 156 for setting the play area. This is because, when setting a play area, the play area control unit 264 first detects the floor surface in the 3D map it generates, and then uses that as a reference to detect walls, furniture, and other objects around the user.

[0062] Furthermore, when acquiring state information of the head-mounted display 100, the cropping unit 274 crops a predetermined-size area 152c that corresponds approximately in front of the head-mounted display 100 and supplies it to the state information acquisition unit 276 as an image 158 for acquiring state information. "Approximately in front" refers to a predetermined-size area that includes, for example, the user's line of sight. The image 158, which includes an image of an object approximately in front of the head-mounted display 100, is less likely to include images of uniform surfaces such as the floor or ceiling, and is more likely to have many feature points. This allows for the continuous acquisition of state information of the head-mounted display 100 with stable accuracy.

[0063] In this way, by optimizing the position and size of the cropping area according to the purpose, multiple pieces of information can be accurately acquired even using only the stereo camera 110. Furthermore, even if the field of view of the stereo camera 110 is widened in anticipation of display, the size of the image to be processed is not affected by the play area settings or the acquisition of the head-mounted display 100 status, thus avoiding unnecessary processing time and achieving both high accuracy and low latency.

[0064] Figure 9 illustrates a more detailed example of the cropping area. In Figures (a) and (b), the pairs of solid rectangles shown represent pairs of stereo image frames (left image 160a, right image 160b). Of these, (a) shows areas 162a and 162b that are cropped for play area setting, and (b) shows areas 164a and 164b that are cropped for acquiring state information of the head-mounted display 100.

[0065] This example assumes that the optical axes 172a and 172b of the two cameras 170a and 170b constituting the stereo camera 110 are mounted in a way that does not coincide with the user's line of sight, which is directed toward the foreground of the figure, as shown on the right side of the figure. Specifically, it is assumed that the optical axes 172a and 172b are mounted on the head-mounted display 100 so that they are angled outward horizontally at a 30° angle to each other, and both are angled 35° downward from the horizontal plane. On the other hand, in Visual SLAM, parallel optical axes are required to identify the position of a point on the surface of the subject by stereo matching.

[0066] Therefore, the optimal regions 162a and 162b for setting the play area in (a) are the inward-facing areas of the original frame pair, left image 160a and right image 160b, in the horizontal direction. In other words, in left image 160a, the optimal crop region is region 162a, which is shifted to the right of the center, and in right image 160b, the optimal crop region is region 162b, which is shifted to the left of the center. On the other hand, in the vertical direction, regions 162a and 162b are located approximately in the center of left image 160a and right image 160b, respectively.

[0067] This is because the optical axes of cameras 170a and 170b, 172a and 172b, are angled slightly downward, making it more likely that the floor surface, which is important for setting the play area, will be captured near the vertical center. In addition, by excluding the area near the bottom edge of left image 160a and right image 160b from regions 162a and 162b, the user's own body is excluded from being reflected in the image.

[0068] In case (b), the optimal regions 164a and 164b for acquiring state information are, in the horizontal direction, the range closer to the inside of the original frame pair, left image 160a and right image 160b. On the other hand, in the vertical direction, regions 164a and 164b are shifted to the upper part of the center of left image 160a and right image 160b. This is because the optical axes 172a and 172b of cameras 170a and 170b are pointed downwards, so objects in front of the head-mounted display 100 are likely to appear in a range that is shifted somewhat upwards.

[0069] In the diagram, as an example, the original frame pair (left image 160a, right image 160b) has a pixel count of 1016×1016 each, and the top-left and bottom-right position coordinates of the cropping areas 162a, 162b, 164a, and 164b are shown with the top-left corner of the image as the origin. Specifically, when setting the play area in (a), a rectangular area 162a is cropped from the left image 160a with position coordinates (294,188) and (933,827) as the top-left and bottom-right vertices, respectively, and a rectangular area 162b is cropped from the right image 160b with position coordinates (82,188) and (721,827) as the top-left and bottom-right vertices, respectively.

[0070] When acquiring state information in (b), a rectangular region 164a is cropped from the left image 160a with position coordinates (294,82) and (933,721) as the top-left and bottom-right vertices, respectively, and a rectangular region 164b is cropped from the right image 160b with position coordinates (82,82) and (721,721) as the top-left and bottom-right vertices, respectively.

[0071] The crop rule memory unit 278 stores information such as position coordinates representing the area to be cropped, optimized for each application, in association with the application. When optimizing the area, the area in which the necessary image will be captured may be calculated based on the orientation of the optical axes 172a and 172b of cameras 170a and 170b as shown in the figure, or the area in which the necessary image is most likely to be captured may be determined experimentally.

[0072] The cropping unit 274 crops the area corresponding to the intended use and sequentially supplies the data in accordance with requests from the play area control unit 264 and the state information acquisition unit 276. The cropped data may also be requested by the app execution unit 290 or the display control unit 292. This allows for efficient display using only the necessary area of ​​the captured image.

[0073] The mounting configuration of the stereo camera 110, the resolution and size of the captured image frame, and the position and size of the crop area shown in the figure are merely examples and are not intended to limit this embodiment. For example, in the figure, the crop area is fixed considering the high probability of the floor surface being captured, but the crop section 274 may be adaptively determined based on the situation at each point in time and the actual image. For example, the range of the area where the floor surface is captured tends to differ between applications that are intended for viewing while seated and applications that are intended for viewing while standing, with the former tending to cover a wider area.

[0074] Therefore, the cropping unit 274 obtains the characteristics of the application being executed from the App execution unit 290, and for applications where the user is assumed to be sitting, it may widen the cropping area or shift it upwards on the image plane compared to applications where the user is assumed to be standing. This increases the likelihood that the floor and objects on it will be captured simultaneously, regardless of the user's posture. The information obtained as application characteristics is not limited to the user's posture; it can also utilize gaze tendencies, such as whether the user tends to look straight ahead or downwards when running the application.

[0075] In these cases, the crop rule storage unit 278 stores information such as the characteristics of the application and the position coordinates representing the area to be cropped, associating them with each application. The crop unit 274 may also use the measurements from the motion sensor 134 of the head-mounted display 100 to determine the area to be cropped. In this case, the crop unit 274 may derive areas that are likely to show the floor surface at each time point based on the orientation of the head-mounted display 100 obtained from these measurements, and change the area to be cropped over time according to the results.

[0076] Alternatively, the cropping unit 274 may initially set the size of the area to be cropped to a small size in the initial stage of setting the play area, and then expand the cropping area to include the image of the floor surface once the floor surface is detected. This allows for the efficient and more comprehensive collection of information about the floor surface and objects on it. For example, the cropping unit 274 expands the cropping area when the state change of the head-mounted display 100 per unit time is smaller than a threshold. This makes it easier to obtain a lot of information, such as feature points, in a short amount of time.

[0077] Once the floor surface is detected, the cropping unit 274 estimates the user's posture, such as whether they are sitting or standing, based on its height, and may adjust the size and position of the cropping area as described above. In these cases as well, the crop rule storage unit 278 stores the rules for changing the cropping area and the conditions used for them in association. Note that the more analysis performed on a single frame, the longer it takes to obtain the results. Therefore, in this embodiment, the basic principle is to obtain results with low latency by fixing or estimating areas where it is highly likely that the necessary image can be obtained, such as the floor surface.

[0078] Figure 9 shows an example where one region is cropped from the left image 160a and the right image 160b for a single application, but the number of regions cropped from each image is not limited. For example, if a camera capable of capturing a panoramic image is introduced, cropping regions within discrete angular ranges will yield an image similar to that obtained by rotating the camera and capturing the surroundings. Therefore, the cropping unit 274 may, for example, crop multiple regions determined by a predetermined rule from a single frame.

[0079] In this case, the cropping unit 274 may select multiple regions with a predetermined number or more of feature points as the cropping targets. The cropping unit 274 may determine the region to be cropped in the current frame by estimating the change in the position of the region to be cropped, which was initially determined by such rules, based on changes in the state of the head-mounted display 100 up to that point.

[0080] Next, the operation of the image display system with the above configuration will be explained. Figure 10 is a flowchart showing the procedure by which the image generation device 200 performs various processing using the captured image. Note that the flowchart shown focuses on the case distinctions of processing by the cropping unit 274 and does not specify the processing order of play area setting or application execution. The flowchart shown is started at any time when the image generation device 200 and the head-mounted display 100 have established communication and at least the captured image data has been transmitted from the head-mounted display 100.

[0081] The image acquisition unit 262 of the image generation device 200 acquires data of the frame to be processed (S10). If the mode for realizing video see-through is activated at this point (Y in S12), the display control unit 292 generates a display image for video see-through by performing necessary correction processing on the target frame and transmits it to the head-mounted display 100 (S14). As described above, the image generation device 200 maintains video see-through mode not only before and after the application execution period and during periods when the user has left the play area, but also during periods when the user is looking around to set the play area.

[0082] The user can select to initialize or reset the play area from the system settings menu of the head-mounted display 100. If initial initialization or resetting of the play area is selected, the play area control unit 264 displays a message on the head-mounted display 100 via the display control unit 292 prompting the user to look around. When the mode for setting the play area is set in this way (Y in S16), the crop unit 274 crops the area for setting the play area from the frame (S18). The play area control unit 264 sequentially acquires the data of the cropped frame and uses it to set the play area (S20).

[0083] As a result, the frame acquired in S10 is used as the display image and is also cropped and used for setting the play area. If it is not in the mode for setting the play area (N in S16), the processes in S18 and S20 are skipped. On the other hand, in S12, if it is not in the mode for realizing video see-through (N in S12), that is, during the period when the application image should be displayed, the cropping unit 274 crops the area for acquiring the state information of the head-mounted display 100 from the frame (S22).

[0084] The state information acquisition unit 276 sequentially acquires data from the cropped frames and uses it to acquire state information of the head-mounted display 100 using Visual SLAM (S24). This state information is sequentially supplied to the App execution unit 290 and used to generate the display image, as well as for the play area control unit 264 to monitor the positional relationship between the user and the play area boundary in order to warn the user.

[0085] In any case, these images are transmitted to the head-mounted display 100 by the display control unit 292 and displayed (S26). The process from S10 to S26 is repeated until a predetermined termination condition is met, such as when the user stops execution (N in S28). When the predetermined termination condition is met (Y in S28), the image generation device 200 terminates all processing.

[0086] Figure 11 is a flowchart showing the processing procedure in which the play area control unit 264 sets the play area in S20 of Figure 10. First, the play area control unit 264 obtains the cropped frame data for setting the play area from the cropping unit 274 (S40). Based on the cropped frame data, the play area control unit 264 automatically detects the play area in the space surrounding the user (S42).

[0087] Specifically, the play area control unit 264 estimates the three-dimensional shape of the user's room based on the cropped image using a known method such as Visual SLAM. In the case of Visual SLAM, this process corresponds to the generation of map data. Based on the estimated three-dimensional shape of the room, the play area control unit 264 detects a plane perpendicular to the direction of gravity indicated by the measurement values ​​of the motion sensor 134 as the floor surface. Furthermore, using this floor surface as a reference, the play area control unit 264 constructs the three-dimensional shapes of objects on it as a set of points corresponding to feature points extracted from the frame.

[0088] The play area control unit 264 determines the boundaries of the play area based on the set of points and generates play area data including its position coordinates. When detecting a play area, the play area control unit 264 derives the height of the floor surface as the play area. The height of the floor surface may be, for example, the distance between the floor surface and the head-mounted display 100 in the direction of gravity. The play area control unit 264 includes data indicating the height of the floor surface in the play area data. If the position of the head-mounted display 100 is the origin, the height of the floor surface may be minus 1 meter, for example.

[0089] The play area control unit 264 checks whether all the data related to the three-dimensional space necessary for setting the play area has been acquired. If it has not been acquired, it repeats the process in S42 for the new frame (N in S46, S40, S42). The data required here is the data necessary to set the play area without any gaps, such as map data that covers the directions the user may face and the directions in which movement is permitted. The play area control unit 264 may also make the determination in S46 by checking the distribution of the state of the stereo camera 110 from which keyframes have been obtained.

[0090] Once the necessary data has been acquired (Y in S46), the play area control unit 264 stores the map data and keyframe data acquired so far in the map storage unit 258 (S48). Subsequently, the play area control unit 264 accepts user input to adjust the play area (S50). For example, the play area control unit 264 generates a floor adjustment screen based on the detected data indicating the height of the floor surface. The floor adjustment screen may include an AR image in which an object representing the floor surface (for example, a semi-transparent grid-like object) is superimposed on the frame of the captured image acquired by the captured image acquisition unit 262.

[0091] The play area control unit 264, via the display control unit 292, causes the floor adjustment screen to be displayed on the display panel of the head-mounted display 100. The play area control unit 264 receives user input to adjust the height of the floor surface via the floor adjustment screen and changes the height of the floor surface accordingly. The play area control unit 264 also generates a play area editing screen based on the detected play area data. The play area editing screen includes an AR image in which an object representing the play area is superimposed on a captured image acquired by the captured image acquisition unit 262.

[0092] The play area control unit 264 displays the play area editing screen on the display panel of the head-mounted display 100. The play area control unit 264 receives user operations to edit the play area from the play area editing screen and changes the shape of the play area 62 accordingly. The play area control unit 264 then stores the final determined play area data in the play area storage unit 256 (S52). The play area data includes, for example, the coordinate values ​​of the point cloud that constitutes the boundary.

[0093] Figure 12 schematically illustrates the user's actions when setting up a play area. Within room 30, the user moves around while looking around with a head-mounted display 100 on their head. In Figure 12, user 32 indicates the user's position at this time. Arrow 34 indicates the direction of the user's gaze. Play area 36 shows an example of a play area that can be set up.

[0094] During processing S40 and S42 in Figure 11, the play area control unit 264 displays the entire frame of the captured image acquired by the captured image acquisition unit 262 on the display panel of the head-mounted display 100 via the display control unit 292, thereby achieving video see-through. As shown in the figure, the user faces in various directions, and the captured images are transmitted from the head-mounted display 100, allowing the play area control unit 264 to construct a three-dimensional map data of the user's surroundings. The output of the motion sensor 134, which is measured simultaneously, may also be used to construct this data.

[0095] Figure 13 illustrates the image of a frame captured by the stereo camera 110 and the keyframe data obtained therefrom. In reality, a pair of images captured by the stereo camera 110 is obtained for each frame, but only one of them is schematically shown in the figure. The play area control unit 264 and the state information acquisition unit 276 extract multiple feature points 44 contained in the frame image 40 using known methods such as the corner detection method.

[0096] The frame image 40 to be processed here is an image cropped by the cropping unit 274 for setting the play area or acquiring state information. As described above, by comparing this with the frame image from the previous time and obtaining corresponding points, the state information of the stereo camera 110 at the time the frame image 40 was taken can be identified. By cropping only the area estimated to be necessary from the original frame and then extracting feature points, the efficiency of these processes can be significantly improved.

[0097] Meanwhile, the play area control unit 264 and the state information acquisition unit 276 select a keyframe 42 if a predetermined number of feature points, for example, 24 or more, are extracted from a single frame image 40. The play area control unit 264 and the state information acquisition unit 276 generate keyframe 42 data consisting of the position coordinates of the feature points (for example, feature points 44) extracted from the selected frame image 40 in the image plane, and an image of a predetermined range around them. The play area control unit 264 and the state information acquisition unit 276 further associate the state information of the stereo camera 110 at the time of capture to create the final keyframe data.

[0098] The play area control unit 264 stores multiple criteria (hereinafter referred to as "bins") for deriving a score regarding the proportion of the user's surrounding space covered by the input captured image (in this embodiment, keyframes generated from the captured image). The multiple bins are multiple criteria that divide the user's surrounding space in different ways.

[0099] Figure 14 illustrates several bins that can be used in this embodiment. Bin 50, Bin 52, and Bin 54 each divide the space around the user, with the user's position at its center and the direction of gravity as the axis. Bin 50 is a reference that divides the space around the user into four sections with a central angle of 90 degrees, based on the user's line of sight at the start of setting up the play area and the direction perpendicular to the line of sight. Bin 52 is shifted in phase by 45 degrees relative to Bin 50. Bin 54 is a reference that divides the space around the user into sixteen sections with a central angle of 22.5 degrees. Hereinafter, the divided areas in the space around the user defined by Bin 50, Bin 52, and Bin 54 will be referred to as "divided areas".

[0100] The play area control unit 264 derives a score based on bins 50, 52, and 54 regarding the proportion of the user's surrounding space covered by the input frame image (in this embodiment, a keyframe generated from the captured image). For example, if the score exceeds a predetermined threshold, the play area control unit 264 determines in S46 of Figure 11 that the data necessary for setting the play area has been acquired.

[0101] Specifically, when the play area control unit 264 generates a keyframe from the captured image, it identifies a division area for each of bins 50, 52, and 54 that matches the state of the head-mounted display 100 at the time the keyframe was obtained. If the obtained keyframe is the first keyframe for that division area, it considers that division area to be filled and adds the number of filled division areas to the score. When the play area control unit 264 generates a new keyframe, it adds the number of newly filled division areas in each bin to the score using a similar algorithm.

[0102] The play area control unit 264 then determines, for example, that the necessary data has not been acquired if the score is less than the threshold "10", and continues the play area detection process. When the score reaches the threshold "10" or higher, the play area control unit 264 determines that the necessary data has been acquired, and moves to the stage where user adjustments are accepted. The score threshold "10" is a value that can be achieved if the user wearing the head-mounted display 100 looks around 180 degrees, but cannot be achieved if the user does not look around.

[0103] Furthermore, the wider the field of view of the stereo camera 110, the more information is contained in one frame, allowing data necessary for play area detection to be acquired with fewer frames. As described above, if the cropping unit 274 crops multiple areas with different angular ranges from the wide-angle image, the user can complete the play area setting with minimal effort required to look around. In this embodiment, since only the cropped area is analyzed, widening the field of view of the stereo camera 110 in this way minimizes the impact on analysis time.

[0104] Figure 15 shows an example of a play area editing screen. The play area editing screen 60 includes a play area 62 and a boundary 64. The play area 62 is an image representing the play area (typically an unobstructed floor surface), and may also be an image representing, for example, a semi-transparent grid-like object. The boundary 64 is an image representing the boundary of the play area 62, and is an image that intersects the play area perpendicularly at the boundary of the play area 62. The boundary 64 may also be, for example, a semi-transparent grid-like object.

[0105] The play area control unit 264 acquires user operations such as moving the boundary 64 or expanding or shrinking the play area 62 on the play area editing screen via the controller 140 or the like. When the user confirms the operation, the play area control unit 264 generates data representing the play area at that point in time as the final state and stores it in the play area storage unit 256.

[0106] According to the embodiment described above, images captured by the stereo camera of the head-mounted display are used for display purposes such as video see-through and augmented reality, as well as for image analysis. In this case, the optimal area is cropped and used according to rules set according to the content and purpose of the analysis. This allows analysis to be performed without reducing efficiency, even if the field of view of the stereo camera is widened. By using this, it is also possible to reduce the effort required for the user to look around when setting up the play area by employing a stereo camera with a wide field of view.

[0107] For example, the image includes the floor when setting the play area, and the image in front of the head-mounted display is used to acquire status information from the head-mounted display. The appropriate area is cropped according to the application, taking into account the camera's optical axis. This allows resources to be concentrated on important areas, resulting in high-precision information being obtained with low latency. Furthermore, since there is no need for dedicated sensors to acquire various types of information, high-quality image representation can be achieved even with a simple head-mounted display configuration. At the same time, it avoids the deterioration of head-mounted display comfort due to increased weight and power consumption.

[0108] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of symbols]

[0109] 10 Image display system, 100 Head-mounted display, 200 Image generation device, 256 Play area memory unit, 258 Map memory unit, 262 Captured image acquisition unit, 264 Play area control unit, 274 Crop unit, 276 Status information acquisition unit, 278 Crop rule memory unit, 292 Display control unit.

Claims

1. a display control unit that causes a moving image being captured by a camera mounted on the head mounted display to be displayed as a display image on the head mounted display; a cropping unit that cuts out a part of an area from a frame of the moving image in accordance with a predetermined rule; an image analysis unit that analyzes an image of the partial area and acquires data necessary for displaying an image different from the moving image on the head mounted display; Equipped with the image analysis unit uses at least the image of the partial area to acquire three-dimensional information of a space around the user, the three-dimensional information being used to set a play area in which the user wearing the head mounted display can move; The information processing apparatus, wherein the cropping unit crops out at least an area from which an image necessary for acquiring the three-dimensional information is likely to be obtained.

2. The information processing apparatus according to claim 1 , wherein the cropping unit crops out an area where it is highly likely that an image of a floor surface will be obtained as the required image.

3. 3 . The information processing apparatus according to claim 1 , wherein the cropping unit estimates a posture of the user and changes the region to be cropped as the region with high possibility in accordance with the result of the estimation. 4 .

4. The image analysis unit acquires a position of a floor surface in the three-dimensional information based on gravity direction information obtained by a motion sensor built into the head mounted display, The information processing apparatus according to claim 2 , wherein the cropping unit determines an area in which an image of the floor surface is likely to be obtained based on a position of the floor surface.

5. The information processing apparatus according to claim 4 , wherein the cropping unit varies the cropped area over time by deriving an area where an image of the floor surface is likely to be obtained at each time based on the position of the floor surface.

6. 6. The information processing apparatus according to claim 1, wherein the cropping section expands the partial area when a change in state of the head mounted display per unit time is smaller than a threshold value.

7. 7. The information processing apparatus according to claim 1, wherein the cropping section cuts out a different area according to the content of the analysis performed by the image analysis section in accordance with a rule associated with the content of the analysis.

8. The information processing apparatus according to claim 7 , wherein the cropping unit crops out a fixed area that is set based on the direction of the optical axis of the camera in accordance with the content of the analysis.

9. the cropping unit cuts out an area that is likely to include many feature points according to the predetermined rule; 9. The information processing apparatus according to claim 1, wherein the image analysis unit further tracks a position and orientation of the head mounted display using the image of the cut-out area.

10. 10. The information processing apparatus according to claim 1, wherein the cropping section crops out a plurality of regions from the frame for one type of analysis performed by the image analysis section.

11. A step of displaying a moving image being captured by a camera mounted on the head mounted display as a display image on the head mounted display; A step of cutting out a part of an area from a frame of the moving image in accordance with a predetermined rule; A step of analyzing an image of the partial area and acquiring data necessary for displaying an image different from the moving image on the head mounted display; Including, The step of acquiring data includes acquiring, using at least the image of the partial area, three-dimensional information of a space around the user, which is used to set a play area in which the user wearing the head-mounted display can move; The information processing method according to the present invention, wherein the cutting out step cuts out at least an area that is likely to provide an image necessary for obtaining the three-dimensional information.

12. A function of displaying a moving image being captured by a camera mounted on the head mounted display as a display image on the head mounted display; A function of extracting a part of an area from a frame of the moving image in accordance with a predetermined rule; a function of analyzing an image of the partial area and acquiring data necessary for displaying an image different from the moving image on the head mounted display; This is realized on a computer. The function of acquiring the data includes at least acquiring, using the image of the partial area, three-dimensional information of a space around the user, which is used to set a play area in which the user wearing the head mounted display can move; The computer program is characterized in that the cutting-out function is at least a region that is likely to provide an image necessary for obtaining the three-dimensional information.

Images