Information processing device and image generation method

The information processing device addresses user discomfort by dynamically positioning information elements on HMDs based on the user's line of sight, ensuring clear visibility and improved interaction within virtual reality environments.

WO2024241414A1PCT designated stage expired Publication Date: 2024-11-28SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/018891
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-22
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Users wearing head-mounted displays (HMDs) face difficulties in confirming their surroundings, especially when displaying two-dimensional images, leading to discomfort due to inadequate presentation methods.

Method used

An information processing device that generates real-space images on an HMD by acquiring the user's line of sight and dynamically changing the position of information elements based on changes in the reference direction, ensuring they remain visible and unobstructive during use.

Benefits of technology

Enhances user experience by providing clear and unobstructed visibility of surroundings and guidance on using input devices, reducing discomfort and improving interaction with virtual reality environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023018891_28112024_PF_FP_ABST
    Figure JP2023018891_28112024_PF_FP_ABST
Patent Text Reader

Abstract

An image generation unit 240 generates a real space image to be displayed on a head-mounted display that is mounted on the head of a user. A reference direction acquisition unit 230 acquires a reference direction related to the line of sight of the user. An image generation unit 240 places an information element in the real space image, at a position derived on the basis of the reference direction. After placing the information element, the image generation unit 240 changes the position of the information element in the real space image if a state, in which the reference direction has changed by a predetermined angle or more with respect to the reference direction at the time of arranging the information element, has continued for a predetermined period of time.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device and image generating method

[0001] The present disclosure relates to a technology for generating an image to be displayed on a head-mounted display (hereinafter also referred to as an "HMD").

[0002] In recent years, users have been wearing an HMD on their head and playing games by operating a game controller while viewing game images in a three-dimensional virtual reality space displayed on the HMD. By performing tracking processing on the HMD and linking the game images to the user's head movements, not only can the sense of immersion in the image world be enhanced, but the entertainment value of the game can also be improved.

[0003] Patent Literature 1 discloses an information processing device that presents a flat image (two-dimensional image) to a user wearing an HMD. This information processing device has a first display mode that always displays the flat image in a direction directly in front of the user, and a second display mode that displays the flat image in a manner that changes in response to changes in the position and / or orientation of the HMD, and has a function for switching between the first display mode and the second display mode based on given conditions.

[0004] International Publication No. 2017 / 051564

[0005] The present inventors have experimented with various image displays in an HMD and have found that the way images are presented can make it difficult for a user to ascertain the current situation. For example, when a two-dimensional image dialogue is displayed on an HMD, they have found that the way it is presented can cause discomfort to the user. Based on the findings obtained through various trials, the present inventors have come up with a display control method suitable for an HMD.

[0006] Therefore, an object of the present disclosure is to provide a technology for effectively generating an image to be displayed on an HMD.

[0007] An information processing device of one aspect of the present disclosure includes one or more processors having hardware, and the one or more processors generate a real-space image to be displayed on a head-mounted display worn on the user's head, obtain a reference direction related to the user's line of sight, place an information element in the real-space image at a position derived based on the reference direction, and, after placing the information element, change the position of the information element in the real-space image if the reference direction continues to change by more than a predetermined angle from the reference direction when the information element was placed for a predetermined period of time.

[0008] Another aspect of an image generation method of the present disclosure includes the steps of generating a real-space image to be displayed on a head-mounted display worn on a user's head, obtaining a reference direction related to the user's line of sight, placing an information element in the real-space image at a position derived based on the reference direction, and, after placing the information element, changing the position of the information element in the real-space image when the reference direction has changed by more than a predetermined angle from the reference direction at the time the information element was placed and this state continues for a predetermined period of time.

[0009] In addition, any combination of the above components, or conversion of the expression of the present disclosure between a method, an apparatus, a system, a computer program, a recording medium on which a computer program is readably recorded, a data structure, etc., is also valid as an aspect of the present disclosure.

[0010] 1 is a diagram illustrating an example of the configuration of an information processing system in an embodiment; FIG. 2 is a diagram illustrating an example of the external shape of an HMD; FIG. 3 is a diagram illustrating functional blocks of an HMD; (a) shows the shape of an input device for a left hand, and (b) shows the shape of an input device for a right hand; FIG. 4 is a diagram illustrating the shape of an input device for a right hand; FIG. 5 is a diagram illustrating the hardware configuration of an information processing device; (b) is a diagram illustrating functional blocks of an information processing device; (c) is a diagram illustrating an example of a real space in which a user exists; (d) is a diagram illustrating an example of a surrounding image; (e) is a diagram illustrating examples of information elements; (f) is a diagram for explaining a method for deriving the placement positions of information elements; (f) is a diagram illustrating examples of placement positions of information elements in a virtual space; (g) is a diagram illustrating examples of information elements; (h) is a diagram illustrating a flowchart for determining the placement positions of information elements; (i) is a diagram illustrating examples of placement positions of information elements in a virtual space; (ii) is a diagram illustrating examples of placement positions of information elements in a virtual space;

[0011] 1 shows an example of the configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes an information processing device 10, a head-mounted display (HMD) 100 that a user wears on his / her head, and an input device 16 that the user holds in his / her hand and operates with his / her fingers. The information processing device 10 executes game software and supplies game image data and audio data to the HMD 100. The information processing device 10 and the HMD 100 may be connected via a known wireless communication protocol or may be connected by a cable.

[0012] The HMD 100 is a display device worn by a user on the head that displays images on a display panel located in front of the user's eyes, and is also called a VR headset. The HMD 100 separately displays an image for the left eye on the left-eye display panel and an image for the right eye on the right-eye display panel. These images form parallax images seen from left and right viewpoints, realizing stereoscopic vision. Because the user views the display panel through optical lenses, the information processing device 10 supplies the HMD 100 with parallax image data that has been corrected for optical distortion caused by the lenses.

[0013] The information processing device 10 and the input device 16 may be connected via a known wireless communication protocol or via a cable. The input device 16 has a plurality of operation members such as operation buttons, and a user operates the operation members with their fingers while holding the input device 16. When the information processing device 10 executes a game, the input device 16 is used as a game controller. The input device 16 has an inertial measurement unit (IMU) including a three-axis acceleration sensor and a three-axis angular velocity sensor, and transmits sensor data to the information processing device 10 at a predetermined frequency (e.g., 800 Hz).

[0014] In order to track the position and orientation of the input device 16 in real space, the input device 16 is provided with a plurality of markers (light emitting units) that can be photographed by the imaging device 14. The information processing device 10 has a function of analyzing the photographed image of the input device 16 and estimating the position and orientation of the input device 16 in real space.

[0015] The HMD 100 is equipped with multiple image capture devices 14. The multiple image capture devices 14 are attached to the front of the HMD 100 at different positions and in different orientations so that the combined capture range of each captures the entire user's field of view. The image capture devices 14 include image sensors capable of capturing images of multiple markers on the input device 16. For example, if the markers emit visible light, the image capture devices 14 include visible light sensors used in general digital video cameras, such as charge-coupled device (CCD) sensors or complementary metal oxide semiconductor (CMOS) sensors. If the markers emit invisible light, the image capture devices 14 include invisible light sensors. The multiple image capture devices 14 capture images of the area in front of the user at a predetermined frequency (e.g., 120 frames per second) in a synchronized manner and transmit image data of the real space to the information processing device 10.

[0016] 2 shows an example of the external shape of the HMD 100. The HMD 100 is composed of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that, when worn by a user, goes around the head and secures the HMD 100 to the head. The wearing band 106 is made of a material or has a structure that allows the length to be adjusted to fit the user's head circumference.

[0017] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the HMD 100 is worn by the user, and includes a display panel inside that faces the eyes when worn. The display panel may be a liquid crystal panel, an organic EL panel, or the like. The housing 108 also includes a pair of optical lenses, one on each side, that are positioned between the display panel and the user's eyes and expand the user's field of view. The HMD 100 may also include speakers or earphones at positions corresponding to the user's ears, and may be configured to allow external headphones to be connected.

[0018] Multiple image capture devices 14a, 14b, 14c, and 14d are provided on the front outer surface of the housing 108. With respect to the direction of the user's face as a reference, image capture device 14a is attached to the upper right corner of the front outer surface so that its optical axis faces diagonally upward to the right, image capture device 14b is attached to the upper left corner of the front outer surface so that its optical axis faces diagonally upward to the left, image capture device 14c is attached to the lower right corner of the front outer surface so that its optical axis faces diagonally downward to the right, and image capture device 14d is attached to the lower left corner of the front outer surface so that its optical axis faces diagonally downward to the left. By installing multiple image capture devices 14 in this manner, the total image capture range, which is the sum of the image capture ranges of each, covers the entire field of view of the user.

[0019] FIG. 3 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data, such as image data, audio data, and sensor data, as well as commands. The memory unit 122 temporarily stores data and commands processed by the control unit 120. The IMU 124 acquires sensor data related to the movement of the HMD 100. The IMU 124 may include at least a three-axis acceleration sensor and a three-axis angular velocity sensor. The IMU 124 detects the values ​​of each axial component (sensor data) at a predetermined frequency (e.g., 800 Hz). The proximity sensor 110 detects whether the user is wearing the HMD 100. The infrared camera 112 is provided in the housing 108 and captures an image of the user's eyes to track the user's gaze.

[0020] The communication control unit 128 transmits data output from the control unit 120 to the external information processing device 10 via a network adapter or an antenna, by wired or wireless communication. The communication control unit 128 also receives data from the information processing device 10 and outputs it to the control unit 120.

[0021] When the control unit 120 receives image data and audio data from the information processing device 10, it supplies the data to the display panel 130 for display and to the audio output unit 132 for audio output. The display panel 130 is composed of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images is displayed on each display panel. The control unit 120 also causes the communication control unit 128 to transmit the sensor data detected by the IMU 124, the audio data acquired by the microphone 126, the image data captured by the imaging device 14, the sensor data acquired by the proximity sensor 110, and the image data captured by the infrared camera 112 to the information processing device 10.

[0022] FIG. 4A shows the shape of a left-handed input device 16a. The left-handed input device 16a includes a case body 80, multiple operating members 82a, 82b, 82c, and 82d (hereinafter, referred to as "operating members 82" unless otherwise specified) operated by the user, and multiple markers 90 that emit light to the outside of the case body 80. Note that in FIG. 4A, only some of the markers are labeled. The markers 90 may have an emitting portion with a circular cross section. The operating members 82 may include an analog stick that is operated by tilting, a push-button, or the like. The case body 80 includes a grip portion 81 and a curved portion 83 that connects the top and bottom of the case body. The user inserts their left hand into the curved portion 83 to grip the grip portion 81. While gripping the grip portion 81, the user operates the operating members 82a, 82b, 82c, and 82d using the thumb of their left hand.

[0023] FIG. 4( b) shows the shape of a right-handed input device 16 b. The right-handed input device 16 b includes a case body 80, multiple operating members 82 e, 82 f, 82 g, and 82 h (hereinafter, referred to as “operating members 82” unless otherwise specified), which are operated by the user, and multiple markers 90 that emit light to the outside of the case body 80. In FIG. 4( b), only some of the markers are labeled. The operating members 82 may include an analog stick operated by tilting, a push-button, or the like. The case body 80 has a grip portion 81 and a curved portion 83 connecting the top and bottom of the case body. The user inserts their right hand into the curved portion 83 to grip the grip portion 81. While gripping the grip portion 81, the user operates the operating members 82 e, 82 f, 82 g, and 82 h using the thumb of their right hand.

[0024] Fig. 5 shows the shape of the input device 16b for the right hand. The input device 16b has operation members 82i and 82j in addition to the operation members 82e, 82f, 82g, and 82h shown in Fig. 4(b). While holding the grip portion 81, the user operates the operation member 82i with the index finger of the right hand and the operation member 82j with the middle finger. Hereinafter, when there is no particular distinction between the input device 16a and the input device 16b, they will be referred to as the "input device 16."

[0025] The marker 90 is a light-emitting portion that emits light to the outside of the case body 80, and includes a resin portion that diffuses and emits light from a light source such as an LED (Light Emitting Diode) element to the outside on the surface of the case body 80. The marker 90 is photographed by the imaging device 14 and used for tracking processing of the input device 16.

[0026] The information processing device 10 uses images captured by the imaging device 14 for tracking processing of the input device 16 and for SLAM (Simultaneous Localization and Mapping) processing of the HMD 100. In an embodiment, of the images captured by the imaging device 14 at 120 frames per second, grayscale images captured at 60 frames per second may be used for tracking processing of the input device 16, and other full-color images captured at 60 frames per second may be used for processing of simultaneously estimating the self-position of the HMD 100 and creating an environmental map.

[0027] The information processing device 10 can operate the HMD 100 in a "see-through view" mode. In the see-through view mode, the information processing device 10 generates an image of real space in the direction the user is facing based on an image captured by the imaging device 14 and provides the image to the HMD 100, which then displays the image of real space on the display panel 130. This allows the user to see their surroundings while wearing the HMD 100. When a user attempts to play a VR game, it is difficult to wear the HMD 100 while holding the input devices 16 in both hands. Therefore, the user first wears the HMD 100 and then holds the two input devices 16 in both hands. Although the user's field of vision is obstructed when the HMD 100 is worn on the head, the HMD 100 displays the real space image on the display panel 130 in the "see-through view" mode, allowing the user to find and hold the input devices 16 in both hands.

[0028] In the see-through view mode, the information processing device 10 can provide information to the user by arranging information elements such as dialogues in the real-space image displayed on the display panel 130. Since the input device 16 of the embodiment is a game controller with an unconventional shape, the HMD 100 may display information elements in the real-space image to guide the user on how to use the input device 16 when the user uses the input device 16 for the first time or after a long time. Additionally, it may be preferable for the HMD 100 to present some information to the user in the see-through view mode. Through various trial and error processes, the present inventor has devised a method for effectively presenting information elements such as dialogues to the user.

[0029] 6 shows the hardware configuration of the information processing device 10. The information processing device 10 includes a main power button 20, a power ON LED 21, a standby LED 22, a system controller 24, a clock 26, a device controller 30, a media drive 32, a USB module 34, a flash memory 36, a wireless communication module 38, a wired communication module 40, a subsystem 50, and a main system 60.

[0030] The main system 60 includes a main CPU (Central Processing Unit), a memory serving as a main storage device, a memory controller, a GPU (Graphics Processing Unit), etc. The GPU is primarily used for processing the game program. These functions may be configured as a system-on-chip and formed on a single chip. The main CPU has the function of executing the game program recorded in the auxiliary storage device.

[0031] The subsystem 50 includes a sub-CPU, a memory serving as a main storage device, a memory controller, and the like, but does not include a GPU and does not have the function of executing a game program. The number of circuit gates of the sub-CPU is smaller than that of the main CPU, and the operating power consumption of the sub-CPU is smaller than that of the main CPU. The sub-CPU operates even while the main CPU is in standby mode, and its processing functions are limited to keep power consumption low.

[0032] The main power button 20 is an input unit through which a user inputs operations, is provided on the front of the housing of the information processing device 10, and is operated to turn on or off the power supply to the main system 60 of the information processing device 10. The power ON LED 21 lights up when the main power button 20 is turned on, and the standby LED 22 lights up when the main power button 20 is turned off.

[0033] The system controller 24 detects a user's pressing of the main power button 20. When the main power button 20 is pressed while the main power is in the off state, the system controller 24 interprets the pressing operation as an "on instruction," and when the main power button 20 is pressed while the main power is in the on state, the system controller 24 interprets the pressing operation as an "off instruction."

[0034] The clock 26 is a real-time clock that generates current date and time information and supplies it to the system controller 24, the subsystem 50, and the main system 60. The device controller 30 is configured as an LSI (Large-Scale Integrated Circuit) that, like a southbridge, transfers information between devices. As shown in the figure, the device controller 30 is connected to devices such as the system controller 24, media drive 32, USB module 34, flash memory 36, wireless communication module 38, wired communication module 40, the subsystem 50, and the main system 60. The device controller 30 absorbs differences in the electrical characteristics and data transfer speeds of each device and controls the timing of data transfer.

[0035] The media drive 32 is a drive device that is driven by loading a ROM medium 44 on which software such as games and license information are recorded, and that reads programs, data, and the like from the ROM medium 44. The ROM medium 44 is a read-only recording medium such as an optical disk, a magneto-optical disk, or a Blu-ray disk.

[0036] The USB module 34 is a module that connects to an external device via a USB cable. The USB module 34 may be connected to an auxiliary storage device such as an SSD or HDD via a USB cable. The flash memory 36 is an auxiliary storage device that constitutes internal storage. The wireless communication module 38 wirelessly communicates with the input device 16 and the HMD 100 using a communication protocol such as Bluetooth (registered trademark) or IEEE802.11. The wired communication module 40 communicates with an external device (not shown) via a wired connection.

[0037] 7 shows functional blocks of the information processing device 10 that realizes the see-through view function. The information processing device 10 includes a processing unit 200 and a communication unit 202. The processing unit 200 includes an acquisition unit 210, a SLAM processing unit 220, an attitude acquisition unit 222, a reference direction acquisition unit 230, a placement position determination unit 232, an operation acceptance unit 234, an information element generation unit 236, an image generation unit 240, and an image provision unit 242. The acquisition unit 210 includes a captured image acquisition unit 212, a sensor data acquisition unit 214, and an operation information acquisition unit 216.

[0038] The information processing device 10 includes a computer, which executes a program to realize various functions shown in FIG. 7 . The computer includes hardware such as a memory into which the program is loaded, one or more processors that execute the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits, including semiconductor integrated circuits and LSIs, and the multiple electronic circuits may be mounted on a single chip or on multiple chips. The functional blocks shown in FIG. 7 are realized by a combination of hardware and software. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms using hardware alone, software alone, or a combination thereof. Note that FIG. 7 illustrates functions for realizing the see-through view mode using the HMD 100, and does not illustrate functions for realizing tracking processing of the input device 16, etc.

[0039] The communication unit 202 receives operation information of the operation member 82 and sensor data transmitted from the input device 16, and supplies these to the acquisition unit 210. The communication unit 202 also receives captured image data and sensor data transmitted from the HMD 100, and supplies these to the acquisition unit 210.

[0040] The captured image acquisition unit 212 acquires, from the HMD 100, captured images captured by the imaging device 14 and captured images captured by the infrared camera 112. The sensor data acquisition unit 214 acquires sensor data detected by various sensors from the HMD 100, and acquires sensor data detected by the IMU from the input device 16. The operation information acquisition unit 216 acquires operation information obtained by a user operating the operation member 82 of the input device 16. Various processes performed by the information processing device 10 will be described below.

[0041] (SLAM Processing) The captured image acquisition unit 212 acquires full-color images for SLAM processing captured by the imaging device 14 of the HMD 100 and supplies them to the SLAM processing unit 220. The sensor data acquisition unit 214 acquires sensor data detected by the IMU 124 of the HMD 100 and supplies it to the SLAM processing unit 220. The SLAM processing unit 220 simultaneously estimates the self-position of the HMD 100 and creates an environmental map based on the image data supplied from the captured image acquisition unit 212 and the sensor data supplied from the sensor data acquisition unit 214.

[0042] The SLAM processing unit 220 connects three-dimensional data measured from the captured images to create a three-dimensional map in a global coordinate system and estimates the position and orientation of the HMD 100 on the three-dimensional map. This three-dimensional map is a virtual space constructed from the real space included in the captured images, and defines the relative positional relationship between the HMD 100 and objects (e.g., walls, floors, televisions, etc.) that exist in the real space. The SLAM processing unit 220 generates a virtual space corresponding to the real space from the images captured by the imaging device 14, allowing the information processing device 10 to grasp the positional relationship between the user wearing the HMD 100 on their head and surrounding objects, thereby assisting the user in safe gameplay.

[0043] (Image Generation Process) The image generation unit 240 generates a real-space image to be displayed on the HMD 100. In the see-through view mode, the image generation unit 240 uses images captured by the imaging devices 14 to generate a surrounding image with the HMD 100 as the base point as a real-space image. This surrounding image is an image of real space with the line of sight being the direction in which the HMD 100 is facing, and preferably is substantially the same as the real space that the user can see when the HMD 100 is removed from their head (i.e., has the same field of view). In the embodiment, the positional relationship between the four imaging devices 14 and the directions of the camera optical axes are known, so the image generation unit 240 can combine images captured by the multiple imaging devices 14 to generate a surrounding image for the left eye and a surrounding image for the right eye.

[0044] FIG. 8 shows an example of a real space in which a user exists. Note that although the space shown in FIG. 8 is shown as a real space here, it actually corresponds to a virtual space generated by the SLAM processing unit 220. This space is the user's room, and a picture of Mt. Fuji is hanging on the wall directly in front of the user, and a television is installed to the user's right. Two input devices 16 are also placed on the floor. Because the user cannot wear the HMD 100 on their head while holding the input devices 16, the user first wears the HMD 100 and then holds the input devices 16 in their hands.

[0045] FIG. 9 shows an example of a surrounding image displayed on the display panel 130 in the see-through view mode. The image generation unit 240 stitches together images captured by multiple image capture devices 14 to generate a surrounding image for the left eye and a surrounding image for the right eye. In the following description, the images for the right eye and the left eye will not be distinguished from each other. The image generation unit 240 may generate a surrounding image using grayscale images captured at 60 frames per second, or may generate a surrounding image using full-color images captured at 60 frames per second. The image generation unit 240 generates a surrounding image with the line of sight in the direction the HMD 100 is facing, and the image providing unit 242 provides the surrounding image to the HMD 100 via the communication unit 202. Therefore, while wearing the HMD 100, the user can see a surrounding image with the same field of view as when not wearing the HMD 100. For example, when the user turns to the right, a surrounding image including a television is displayed on the display panel 130.

[0046] When the see-through view mode ends and the user starts playing the game, the image generation unit 240 may execute the game program based on the operation information of the user operating the input device 16, and generate a game image.

[0047] (Reference Direction Acquisition Process) The reference direction acquisition unit 230 acquires a reference direction related to the user's line of sight. The reference direction acquisition unit 230 may determine the reference direction depending on the posture of the HMD 100. The posture acquisition unit 222 acquires the posture of the HMD 100 based on sensor data from the IMU 124. At this time, the posture acquisition unit 222 may determine the posture of the HMD 100 using the position of the HMD 100 estimated by the SLAM processing unit 220. Note that if the SLAM processing unit 220 can estimate the posture of the HMD 100, the posture acquisition unit 222 may acquire the posture of the HMD 100 from the SLAM processing unit 220. The reference direction acquisition unit 230 may identify the direction in which the HMD 100 is facing (i.e., the direction in which the user's face is facing) from the posture of the HMD 100 and acquire that direction as the reference direction. The reference direction acquisition unit 230 may acquire, as the reference direction, a direction obtained by changing the direction in which the HMD 100 is facing by a predetermined angle (for example, about 5 degrees) in the vertical direction.

[0048] The reference direction acquisition unit 230 may acquire the user's gaze direction as the reference direction. In this case, the reference direction acquisition unit 230 identifies the user's gaze direction from the image captured by the infrared camera 112 acquired by the captured image acquisition unit 212. For example, the reference direction acquisition unit 230 may acquire a direction between the gaze direction of the left eye and the gaze direction of the right eye as the reference direction.

[0049] (Operation in See-Through View Mode) The following describes the operation of the information processing device 10 in the see-through view mode. For example, when a user uses the input device 16 for the first time, the information processing device 10 may provide the user with information elements that guide the user on how to use the input device 16. By providing the user with information elements in the see-through view mode, the information processing device 10 enables the user to try operating the input device 16 according to the guidance while checking the surrounding situation. Therefore, the following describes an example in which the information processing device 10 synthesizes information elements that guide the user on how to use the input device 16 into a real-space image and provides the information elements to the user.

[0050] The information element generation unit 236 generates information elements for presenting guidance information to the user. The information element generation unit 236 may generate the information elements in the form of a dialogue. The dialogue may be displayed during initial setup when the user uses the input device 16 for the first time, or may be displayed when the user operates a predetermined button (not shown) provided on the HMD 100.

[0051] 10 shows an example of an information element 300. The information element 300 presenting guidance may be generated as a still image or a moving image. In the embodiment, the information element 300 is a dialog having a two-dimensional rectangular shape, but may have another shape. Furthermore, the information element 300 in the embodiment is an opaque dialog, but may also be a translucent dialog.

[0052] The placement position determination unit 232 determines the position at which the information element 300 is to be placed in the virtual three-dimensional space constructed by the SLAM processing unit 220. This virtual three-dimensional space corresponds to the real space and determines the relative positional relationship with respect to the HMD 100. In the embodiment, the placement position determination unit 232 derives the initial position of the information element 300 based on the reference direction acquired by the reference direction acquisition unit 230.

[0053] 11 is a diagram illustrating a method for deriving the placement position of the information element 300. The information element 300 is placed upright in the vertical direction in the virtual space. The placement position determination unit 232 matches the height of the center of the information element 300 to the height H of the HMD 100 in the virtual space. Note that the height H of the HMD 100 may be a height estimated by the SLAM processing unit 220.

[0054] The placement position determination unit 232 also determines the position of the center of the information element 300 to be a predetermined distance L away from the HMD 100 in the direction of the horizontal component of the reference direction. The distance L is set to a distance that is neither too close nor too far from the HMD 100, and may be set to, for example, 1.2 meters. The orientation (posture) of the two-dimensional rectangular information element 300 is set so that it is perpendicular to the direction of the horizontal component of the reference direction. The placement position determination unit 232 provides the determined placement position of the information element 300 to the image generation unit 240.

[0055] 12 shows an example of the placement position of the information element 300 in the virtual space. The image generation unit 240 places the information element 300 in the placement position determined by the placement position determination unit 232 in the virtual space. That is, the image generation unit 240 sets the center of the information element 300 at a position that is a predetermined distance L away from the HMD 100 and at the same height as the HMD 100, and places the information element 300 in a two-dimensional rectangular shape so that it intersects with the horizontal component of a reference direction that has the HMD 100 as its base point. In this embodiment, the information element 300 is placed upright in the vertical direction, and therefore the horizontal component of the reference direction that has the HMD 100 as its base point passes vertically through the center of the information element 300.

[0056] As described above, the image generation unit 240 places the information element 300 in the real space image at a position derived based on the reference direction, and the image provision unit 242 provides the generated image to the HMD 100 via the communication unit 202.

[0057] 13 shows an example of an information element 300 displayed on the display panel 130. The height of the center of the information element 300 is adjusted to the height of the HMD 100, so when the user is facing upward from the horizontal, the information element 300 is displayed on the lower side of the screen of the display panel 130. Therefore, when the user is facing downward from the horizontal, the information element 300 is displayed on the upper side of the screen of the display panel 130.

[0058] 14 shows an example of an information element 300 displayed on the display panel 130. The height of the center of the information element 300 is aligned with the height of the HMD 100, so when the user faces horizontally, the information element 300 is displayed in the center in the height direction of the display panel 130. By aligning the height of the center of the information element 300 with the height of the HMD 100, the user can view the information element 300 in a natural posture facing horizontally.

[0059] As described above, the information element 300 is placed at an initial position derived based on the reference direction, and the display position of the information element 300 is fixed and does not move until a predetermined condition is satisfied. For ease of explanation, the reference direction when the initial position of the information element 300 is derived will be referred to as the "reference direction S." The reference direction acquisition unit 230 acquires the ever-changing reference direction, and the placement position determination unit 232 monitors the change in the reference direction acquired by the reference direction acquisition unit 230 and determines whether the change in the reference direction satisfies a predetermined display position change condition.

[0060] 15 shows a flowchart for determining the placement position of the information element 300. The reference direction acquisition unit 230 acquires the reference direction at a predetermined cycle, and the placement position determination unit 232 specifies the reference direction S at the timing when display of the information element 300 starts (S10). The placement position determination unit 232 derives the placement position of the information element 300 based on the reference direction S (S12), and the image generation unit 240 places the information element 300 in the position derived by the placement position determination unit 232 in the real-space image (S14).

[0061] After the information element 300 is placed, the placement position determination unit 232 monitors changes in the reference direction derived by the reference direction acquisition unit 230. The placement position determination unit 232 determines whether a state in which the reference direction has changed by a predetermined angle or more with respect to the reference direction S when the information element 300 was placed continues for a predetermined period of time (S16). Here, the predetermined angle may be set to approximately half the screen viewing angle of the display panel 130. For example, if the screen viewing angle is 110 degrees, the predetermined angle may be set to 50 degrees. By setting the predetermined angle in this manner, a state in which at least a portion of the information element 300 is not displayed becomes a condition for changing the display position of the information element 300.

[0062] The predetermined angle in the display position change condition may be set as a horizontal angle. Therefore, the placement position determination unit 232 may determine whether or not the state in which the reference direction has changed by more than the predetermined angle in the horizontal direction with respect to the reference direction S continues for a predetermined period of time after the information element 300 is placed.

[0063] The predetermined time in the display position change condition is preferably set to a time shorter than 1 second, for example, 0.8 seconds. After experimenting with various time periods, the present inventors found that if the entire information element 300 is not visible for more than 1 second, the information element 300 is lost for too long, making it seem inappropriate as guidance. Therefore, in this embodiment, if the reference direction is changed by 50 degrees or more horizontally relative to the reference direction S for 0.8 seconds (Y in S16), the placement position determination unit 232 identifies the reference direction acquired by the reference direction acquisition unit 230 at that time as a new reference direction S (S10). The placement position determination unit 232 derives a new position of the information element 300 based on the new reference direction S (S12), and the image generation unit 240 changes the position of the information element 300 in the real-space image (S14). In other words, the image generation unit 240 relocates the information element 300 to a position derived based on the reference direction when the display position change condition is met.

[0064] On the other hand, if the state in which the reference direction has changed by 50 degrees or more relative to the reference direction S does not continue for 0.8 seconds (N in S16), the placement position determination unit 232 will not change the position of the information element 300 in the real-space image and will continue to monitor changes in the reference direction.

[0065] In the see-through view mode, it is necessary for the user to be able to check the surroundings. For example, if the user looks down to find the input device 16 placed on the floor, it would be difficult to find the input device 16 if the information element 300 was always displayed on the display panel 130. Therefore, by fixing the position of the information element 300 unless the display position change condition shown in S16 is satisfied, the user can easily check the surroundings. Furthermore, since the user often looks down when wearing the HMD 100, for example, to check for obstacles on the floor, the height of the information element 300 is set to the height of the HMD 100 so that the information element 300 does not get in the way when the user looks down.

[0066] 16 shows an example of the placement position of the information element 300 in the virtual space. In this example, the user faces to the right (toward the TV) from the state shown in FIG. 12. At this time, the placement position determination unit 232 detects that the reference direction has changed by a predetermined angle (e.g., 50 degrees) or more in the horizontal direction with respect to the reference direction S when the information element 300 was placed. The placement position determination unit 232 does not change the position of the information element 300 until this state continues for a predetermined time (e.g., 0.8 seconds).

[0067] 17 shows an example of the placement position of the information element 300 in the virtual space. When the reference direction is changed by more than a predetermined angle with respect to the reference direction S when the information element 300 is placed, and this state of change in the horizontal direction by more than the predetermined angle continues for a predetermined period of time, the placement position determination unit 232 acquires the reference direction at the time when this state continues for the predetermined period of time as a new reference direction S, and derives a new placement position of the information element 300 based on the new reference direction S. The image generation unit 240 places the information element 300 in the real-space image at the position derived by the placement position determination unit 232, and the image provision unit 242 supplies the generated image to the HMD 100.

[0068] 18 shows an example of an information element 300 displayed on the display panel 130. When the display position change condition is met, the information element 300 is displayed at a new position in the real-space image. Because the information element 300 follows the change in the reference direction with a slight delay, the user can view the information element 300 without a significant delay even if the user moves their head.

[0069] When changing the position of the information element 300, the image generation unit 240 preferably moves the information element 300 from its original position (referred to as "position A") to a new position (referred to as "position B") while changing the movement speed. The movement time is set to, for example, 0.5 to 0.6 seconds. When the display position change condition is met, the image generation unit 240 moves the information element 300 so that the speed at which the information element 300 starts moving from position A is relatively fast and the speed at which the information element 300 stops at position B is relatively slow. After trying various movement modes, the present inventor found that moving the information element 300 from position A to position B at a constant speed may cause motion sickness, and therefore, this problem is solved by changing the movement speed.

[0070] Note that if the distance between the HMD 100 and the information element 300 becomes shorter, the information element 300 will be displayed larger on the display panel 130, making it difficult for the user to confirm the information presented in the information element 300. Therefore, when the HMD 100 approaches the information element 300, the placement position determination unit 232 changes the position of the information element 300 in the real-space image so that the distance between the HMD 100 and the information element 300 does not become closer than a predetermined threshold. As described above, the placement position determination unit 232 determines the position of the information element 300 to be a predetermined distance L (e.g., 1.2 meters) from the HMD 100, and therefore the threshold may be set to a value smaller than the distance L, e.g., 1.1 meters. By limiting the approach distance in this manner, the information element 300 displayed on the display panel 130 can always be easily recognized.

[0071] On the other hand, when the distance between the HMD 100 and the information element 300 increases, the information element 300 is displayed smaller on the display panel 130, allowing the user to view a wider range of real space. Therefore, even if the HMD 100 moves away from the information element 300, the placement position determination unit 232 does not change the position of the information element 300 in the real space image.

[0072] In the see-through view mode, the user can operate the operation member 82 of the input device 16. For example, when the user operates a predetermined operation member 82 while the information element 300 shown in Fig. 18 is being displayed, the operation accepting unit 234 accepts the selection of the "Next" button, and the information element generating unit 236 generates the information element 300 that will be the next guidance. On the other hand, the user may erroneously operate the operation member 82. For example, it is undesirable for the user to erroneously operate the operation member 82 when he or she is not looking at the information element 300, causing the next guidance screen to be displayed.

[0073] Therefore, after the information element 300 is placed, if the reference direction changes by more than a predetermined second angle with respect to the reference direction S when the information element 300 is placed, the operation acceptance unit 234 invalidates the operation of the operation member 82. In other words, the operation acceptance unit 234 ignores the operation of the operation member 82 and does not accept the selection operation of the "Next" button. For example, the predetermined second angle may be smaller than the angle used in the display position change condition. The predetermined second angle may be set separately for the horizontal direction and the vertical direction; for example, the horizontal angle may be set to 38 degrees and the vertical angle may be set to 32 degrees.

[0074] The present disclosure has been described above based on examples. The above examples are merely illustrative, and those skilled in the art will understand that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present disclosure. In the examples, the image generation processing is performed by the information processing device 10, but the functions of the information processing device 10 may be provided in the HMD 100, and the HMD 100 may perform the image generation processing. In other words, the HMD 100 may be the information processing device 10.

[0075] The present disclosure can be used in a technology for generating images to be displayed on an HMD.

[0076] 1: Information processing system, 10: Information processing device, 14: Imaging device, 16: Input device, 82: Operation member, 100: HMD, 112: Infrared camera, 120: Control unit, 122: Memory unit, 124: IMU, 126: Microphone, 128: Communication control unit, 130: Display panel, 132: Audio output unit, 200: Processing unit, 202: Communication unit, 210: Acquisition unit, 212: Captured image acquisition unit, 214: Sensor data acquisition unit, 216: Operation information acquisition unit, 220: SLAM processing unit, 222: Attitude acquisition unit, 230: Reference direction acquisition unit, 232: Placement position determination unit, 234: Operation acceptance unit, 236: Information element generation unit, 240: Image generation unit, 242: Image provision unit, 300: Information element.

Claims

1. An information processing device comprising one or more processors having hardware, the one or more processors generating a real-space image to be displayed on a head-mounted display worn on a user's head, acquiring a reference direction related to the user's line of sight, placing an information element in the real-space image at a position derived based on the reference direction, and, after placing the information element, changing the position of the information element in the real-space image if the reference direction is changed by a predetermined angle or more from the reference direction at the time the information element was placed and this state continues for a predetermined period of time.

2. The information processing device according to claim 1, wherein the one or more processors acquire images captured by an imaging device provided in the head-mounted display, and use the acquired captured images to generate an ambient image based on the head-mounted display as a real-space image.

3. The information processing device according to claim 1, wherein the one or more processors acquire the direction in which the head-mounted display is facing as the reference direction.

4. The information processing device according to claim 1, wherein the one or more processors acquire the user's line of sight as a reference direction.

5. The information processing device according to claim 1, wherein the one or more processors arrange the information elements so as to intersect in the direction of a horizontal component of the reference direction.

6. The information processing device according to claim 5, wherein the one or more processors position the information element at a predetermined distance from the head mounted display.

7. The information processing device according to claim 5, wherein the one or more processors arrange the information elements at a height of a head-mounted display.

8. The information processing device according to claim 1, wherein the one or more processors change the position of the information element in the real space image when, after placing the information element, the reference direction is changed horizontally by a predetermined angle or more from the reference direction when the information element was placed and this state continues for a predetermined period of time.

9. The information processing device according to claim 1, wherein the one or more processors, after placing the information element, if the reference direction changes by a predetermined angle or more from the reference direction when the information element was placed and this state continues for a predetermined period of time, place the information element in a position derived based on the reference direction at the time the predetermined period of time has continued.

10. The information processing device according to claim 1, wherein the one or more processors change the position of the information element in the real space image when the head mounted display approaches the information element so that the distance between the head mounted display and the information element does not become closer than a predetermined threshold.

11. The information processing device according to claim 10, wherein the one or more processors do not change the position of the information element in the real space image even if the head mounted display moves away from the information element.

12. The information processing device according to claim 1, wherein the one or more processors invalidate the operation of the operating member when, after placing the information element, the reference direction changes by a predetermined second angle or more with respect to the reference direction when the information element was placed.

13. An image generating method comprising the steps of: generating a real-space image to be displayed on a head-mounted display worn on a user's head; acquiring a reference direction related to the user's line of sight; arranging an information element in the real-space image at a position derived based on the reference direction; and, after the information element has been arranged, changing the position of the information element in the real-space image when a state in which the reference direction has changed by a predetermined angle or more from the reference direction at the time the information element was arranged continues for a predetermined period of time.

14. A program for enabling a computer to perform the following functions: generating a real-space image to be displayed on a head-mounted display worn on a user's head; obtaining a reference direction related to the user's line of sight; placing an information element in the real-space image at a position derived based on the reference direction; and changing the position of the information element in the real-space image when, after placing the information element, the reference direction has changed by a predetermined angle or more from the reference direction at the time the information element was placed and this state continues for a predetermined period of time.

Citation Information

Patent Citations

  • Matching content to spatial 3D environments

    JP2020520498A

  • Method and device for tailoring a synthesized reality experience to a physical setting

    US20210056749A1

  • Information processing device, information processing method, and program

    WO2019044188A1