Display control apparatus and display control method

Abstract

To prevent a user wearing a transparent wearable device from moving in a direction that departs from an intended direction.SOLUTION: A determination unit 212 determines a line-of-sight direction Dv and a target moving direction Df. The line-of-sight direction Dv is a direction in which a user U wearing MR glasses 30 views a virtual object VO that the user U gazes. The target moving direction Df is a direction in which the user U is to move. When an angle θd formed by the line-of-sight direction Dv and the target moving direction Df is larger than a threshold angle θth, a display control unit 214 moves the virtual object VO to a position where the angle θd formed by the line-of-sight direction Dv and the target moving direction Df decreases.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a display control device and a display control method. [Background technology]

[0002] A user wearing a glasses device such as a head-mounted display (HMD) may walk while viewing content displayed on the HMD. For example, Patent Document 1 discloses an HMD that fixes objects displayed on the display unit to real-world spatial coordinates when the user wearing the HMD is standing still, and fixes objects displayed on the display unit to coordinates based on the user's head when the user is walking. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-82411 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional HMDs, if the direction in which the content being viewed by the user is displayed differs from the direction in which the user should be moving, the user will end up walking while looking in a direction different from the direction in which they should be moving, which can lead to a problem in which the user's actual moving direction may deviate from the direction in which they should be moving.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to prevent the actual direction of travel of a user wearing a transparent wearable device from deviating from the direction the user should be traveling. [Means for solving the problem]

[0006] A preferred embodiment of the present invention provides a display control device that includes a determination unit that determines a first direction of a virtual object that a user wearing a transparent wearable device is gazing at as seen by the user and a second direction in which the user should move, and a display control unit that, when the angle between the first direction and the second direction is greater than a threshold angle, moves the virtual object to a position where the angle between the first direction and the second direction is smaller.

[0007] A preferred aspect of the present invention provides a display control method that determines a first direction in which a user wearing a transparent wearable device gazes at a virtual object, as seen by the user, and a second direction in which the user should move; and if the angle between the first direction and the second direction is greater than a threshold angle, moves the virtual object to a position where the angle between the first direction and the second direction is smaller. [Effects of the Invention]

[0008] The display control device and display control method according to the present invention prevent the actual moving direction of a user wearing a see-through wearable device from deviating from the direction in which the user should move. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of an information processing system including a display control device according to a first embodiment. [Figure 2] 2 is a schematic diagram showing an example of a virtual space displayed by the MR glasses of FIG. 1. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of a server in FIG. 1. [Figure 4] FIG. 2 is a perspective view showing the appearance of the MR glasses of FIG. 1. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of the MR glasses of FIG. 1. [Figure 6] 2 is a block diagram showing an example of the configuration of the terminal device of FIG. 1. [Figure 7] FIG. 10 is a diagram showing an example of an image captured by an imaging device while a user is walking. [Figure 8] FIG. 8 is a diagram showing an example of an image captured a certain time after the image shown in FIG. 7 is captured. [Figure 9] FIG. 10 is a diagram illustrating a positional relationship between the MR glasses and a first virtual object. [Figure 10] FIG. 10 is a schematic diagram showing an example of moving a first virtual object. [Figure 11] FIG. 10 is a schematic diagram showing another example of moving the first virtual object. [Figure 12] 5 is a flowchart showing the operation of the processing device of FIG. 4. [Figure 13] FIG. 10 is a schematic diagram showing an example of the operation of a processing device of a terminal device according to Modification 1. [Figure 14] FIG. 10 is a schematic diagram showing another example of moving the first virtual object. [Figure 15] FIG. 10 is a schematic diagram showing another example of moving the first virtual object. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1. First embodiment The configuration of a display control device according to a first embodiment of the present invention will be described below with reference to FIGS.

[0011] 1.1. Configuration of the First Embodiment 1.1.1. Overall structure 1 is a diagram showing the overall configuration of an information processing system 1 including a display control device according to the first embodiment. The information processing system 1 includes a server 10, a terminal device 20, MR glasses 30, and a communication network NET.

[0012] The server 10 is a device that provides digital content. The server 10 receives requests from the terminal device 20 via the communication network NET, and distributes various digital contents to the terminal device 20 in response to the requests from the terminal device 20.

[0013] The terminal device 20 is a device carried by the user U. In this embodiment, the terminal device 20 includes devices such as a PC, a tablet terminal, a smartphone, and a smartwatch. In this embodiment, the terminal device 20 will be described assuming that it is a smartphone. The terminal device 20 displays various digital contents distributed from the server 10 on the MR glasses 30 as virtual objects. The terminal device 20 is an example of a display control device.

[0014] The MR glasses 30 are a wearable display device worn on the head of the user U. The MR glasses 30 display virtual objects on display panels provided in each lens corresponding to each eye of the user U. Each lens and display panel of the MR glasses 30 are see-through. Therefore, the user U wearing the MR glasses 30 can view the real space as an external world image through each lens and display panel of the MR glasses 30. The external world image may be a virtual image obtained by capturing an image of the user U's surroundings. A method using a real external world image is called an optical see-through method. A method using a virtual external world image is called a video see-through method. In the following explanation, the MR glasses 30 will be described as employing the optical see-through method. The MR glasses 30 are an example of a see-through wearable device.

[0015] In the information processing system 1, the server 10 and the terminal device 20 are connected to each other so as to be able to communicate with each other via a communication network NET. The MR glasses 30 may be connected to the communication network NET and communicate with the terminal device 20 via the communication network NET.

[0016] The information processing system 1 is a system that provides a service of delivering digital content and the like to be displayed in a virtual space to a user U wearing MR glasses 30 on his / her head.

[0017] FIG. 2 is a schematic diagram showing an example of a virtual space VS displayed by the MR glasses 30 of FIG. 1. The virtual space VS is defined as the space inside a virtual celestial sphere CS centered at the position of the user U's head. If the radius of the celestial sphere CS is R, the virtual space VS can be said to be the space inside a sphere of radius R. In other words, the celestial sphere CS represents the entire range of the virtual space VS. The zenith ZE of the celestial sphere CS is a point above the user U's head on the surface SS of the celestial sphere CS. The nadir NA of the celestial sphere CS is a point below the user U's feet on the surface SS. The equator EQ is the 0-degree latitude line of the celestial sphere CS. A plane including the equator EQ and a line connecting the zenith ZE and the nadir NA are mutually perpendicular. Note that the virtual space VS may also be defined as the space inside a cylinder.

[0018] As shown in Fig. 2, one or more virtual objects VO are placed in the virtual space VS. In this example, the virtual objects VO include five virtual objects VO1 to VO5. In this embodiment, each of the virtual objects VO1 to VO5 is placed on a surface SS of the celestial sphere CS. Note that although a user U is shown in Fig. 2 for convenience, the user U is not actually displayed on the MR glasses 30.

[0019] The one or more virtual objects VO are, for example, virtual objects representing data such as still images, videos, three-dimensional CG models, HTML files, and text files, and virtual objects representing applications. Here, examples of text files include memos and source code. Examples of applications include browsers, applications for using SNS, and applications for generating document files. Note that the number of one or more virtual objects VO in FIG. 2 is merely an example, and the number of one or more virtual objects VO is not limited to the present disclosure.

[0020] 1.1.2. Server configuration Fig. 3 is a block diagram showing an example of the configuration of the server 10 in Fig. 1. As shown in Fig. 3, the server 10 includes a processing device 11, a storage device 12, and a communication device 13. The elements included in the server 10 are connected to each other by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, unit, etc.

[0021] The processing device 11 is a processor that controls the entire server 10, and is configured using, for example, one or more chips. The processing device 11 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 11 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 11 executes various processes in parallel or sequentially.

[0022] The storage device 12 is a recording medium that can be read and written by the processing device 11. The storage device 12 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM).

[0023] The storage device 12 stores a plurality of programs including a control program PR1 to be executed by the processing device 11. The storage device 12 also functions as a work area for the processing device 11. The control program PR1 is a program that controls the entire processing device 11.

[0024] The communication device 13 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0025] 1.1.3.Configuration of MR glasses 4 is a perspective view showing the appearance of the MR glasses 30 of FIG. 1. The MR glasses 30 include temples 94L and 94R, a bridge 96, projection optical systems 98L and 98R, and a sound output device 38. In the following description, when distinguishing between similar elements, a suffix such as "L" for temple 94L and "R" for temple 94R is used. When not distinguishing between similar elements, only a common number without a suffix is ​​used, such as temple 94.

[0026] The temples 94 are rod-shaped components supported by the pinna. The bridge 96 is disposed between the projection optical system 98L and the projection optical system 98R. The projection optical system 98 includes a display device 37, a light guide path 981, and a half mirror 982. The bridge 96 is provided with the imaging device 35. The imaging device 35 captures an image of the external environment in front of the user U.

[0027] The display device 37 is disposed within the temple 94. The display device 37 displays an image. The display device 37 has various display panels, such as a liquid crystal panel and an organic EL (Electro Luminescence) panel. When the display device 37 displays an image, light representing the image is emitted from the display device 37. The light emitted from the display device 37 is guided to the half mirror 982 by a light guide path 981. The half mirror 982 reflects the light guided by the light guide path 981. The light reflected by the half mirror 982 is projected onto the retina of the user U. The user U recognizes the image from this light. The half mirror 982 has a surface facing the user. When the display device 37 does not display an image, the user U can see the outside world through the half mirror 982.

[0028] The sound output device 38 is disposed on a side surface of the temple 94. The sound output device 38 outputs sound. In this embodiment, the MR glasses 30 are operated using the terminal device 20, and therefore do not include a particular input device, but may include an input device such as a touch sensor, or may include a sound collection device such as a microphone.

[0029] Fig. 5 is a block diagram showing an example of the configuration of the MR glasses 30 in Fig. 1. As shown in Fig. 5, the MR glasses 30 include a processing device 31, a storage device 32, a line-of-sight acquisition device 33, a movement detection device 34, an imaging device 35, a communication device 36, a display device 37, and a sound output device 38. The elements of the MR glasses 30 are connected to each other by one or more buses for communicating information.

[0030] The processing device 31 is a processor that controls the entire MR glasses 30, and is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 31 may be realized by hardware such as a DSP, ASIC, PLD, or FPGA. The processing device 31 executes various processes in parallel or sequentially.

[0031] The storage device 32 is a recording medium that can be read and written by the processing device 31. The storage device 32 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0032] The storage device 32 stores a plurality of programs including a control program PR3 to be executed by the processing device 31. The storage device 32 also functions as a work area for the processing device 31.

[0033] The gaze acquisition device 33 acquires the direction in which the user is looking, i.e., the direction of the user's gaze, by eye tracking, which tracks the movement of the user U's left and right eyeballs, and outputs gaze direction information indicating the direction of the user U's gaze to the processing device 31 based on the acquired results.

[0034] More specifically, the line-of-sight acquisition device 33 includes a pair of light sources and a pair of cameras. Each of the pair of light sources corresponds to the left eye and the right eye of the user U, and each of the pair of cameras corresponds to the left eye and the right eye of the user U. The light source corresponding to the left eye of the user U irradiates infrared light onto the left eye of the user U, and the camera corresponding to the left eye of the user U captures a corneal reflection image of the left eye of the user U as an image formed by reflection of the irradiated infrared light. Similarly, the light source corresponding to the right eye of the user U irradiates infrared light onto the right eye of the user U, and the camera corresponding to the right eye of the user U captures a corneal reflection image of the right eye of the user U.

[0035] The gaze acquisition device 33 detects the positions of the inner corners of the eyes and the irises of the user U from the acquired corneal reflection images of the left eye and the right eye, and acquires gaze direction information of the user U based on the detected positions of the inner corners of the eyes and the irises of the user U. Note that the method of acquiring the gaze by the gaze acquisition device 33 is not limited to the above method, and any method may be used.

[0036] The motion detection device 34 detects the motion of the MR glasses 30 and outputs the motion information to the processing device 31. The motion information includes acceleration information indicating the acceleration in each of the X-axis, Y-axis, and Z-axis directions, and angular acceleration information indicating the angular acceleration about each of the X-axis, Y-axis, and Z-axis as the center of rotation. The motion detection device 34 includes an acceleration sensor that detects acceleration, an inertial sensor such as a gyro sensor that detects angular acceleration, and a geomagnetic sensor that detects the direction in which the MR glasses 30 are facing.

[0037] The acceleration sensor detects acceleration in each of the mutually orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around each of the X-, Y-, and Z-axes as the center of rotation. The geomagnetic sensor detects geomagnetism in each of the X-, Y-, and Z-axes, and detects the direction in which the MR glasses 30 are facing. The movement detection device 34 outputs movement information related to the movement of the MR glasses 30 to the processing device 31.

[0038] The imaging device 35 outputs imaging information obtained by capturing an image of the external world. The imaging device 35 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 31.

[0039] The communication device 36 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 36 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 36 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 36 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, USB, etc. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0040] The display device 37 is a device that displays images. The display device 37 displays various images under the control of the processing device 31. The display device 37 includes a display panel for the left eye and a display panel for the right eye.

[0041] The sound output device 38 is a device that outputs sound. The sound output device 38 displays various sounds under the control of the processing device 31. The sound output device 38 includes a speaker for the left ear and a speaker for the right ear.

[0042] The processing device 31 functions as an acquisition unit 311, an output unit 312, and a display control unit 313, for example, by reading out a control program PR3 from the storage device 32 and executing it.

[0043] The acquisition unit 311 acquires a control signal and image information from the terminal device 20. The control signal is a signal for controlling display on the display device 37. The control signal is generated by a display control unit 214 (described later) provided in the terminal device 20. The image information is information indicating an image to be displayed on the display device 37. The image information is information including information indicating the virtual objects VO1 to VO5. The image information may be stored in the storage device 22 of the terminal device 20, or may be stored in the storage device 12 of the server 10.

[0044] The acquisition unit 311 also acquires line-of-sight information input from the line-of-sight acquisition device 33, movement information input from the movement detection device , and imaging information input from the imaging device .

[0045] The output unit 312 outputs the line of sight information, movement information, and imaging information acquired by the acquisition unit 311 to the communication device 36.

[0046] The display control unit 313 controls the display on the display device 37 based on the control signal acquired by the acquisition unit 311. Furthermore, the display control unit 313 causes the display device 37 to display the virtual object VO based on the image information acquired by the acquisition unit 311.

[0047] 1.1.4. Terminal Device Configuration Fig. 6 is a block diagram showing an example of the configuration of the terminal device 20 in Fig. 1. As shown in Fig. 2, the terminal device 20 includes a processing device 21, a storage device 22, a communication device 23, a display device 24, an input device 25, and a positioning device 26. The elements included in the terminal device 20 are connected to each other by one or more buses for communicating information.

[0048] The processing device 21 is a processor that controls the entire terminal device 20, and is configured using, for example, one or more chips. The processing device 21 is configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 21 may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 21 executes various processes in parallel or sequentially.

[0049] The storage device 22 is a recording medium that can be read and written by the processing device 21. The storage device 22 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0050] The storage device 22 stores a plurality of programs including a control program PR2 to be executed by the processing device 21. The storage device 22 also functions as a work area for the processing device 21. The control program PR2 is a program that controls the entire processing device 21.

[0051] The communication device 23 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 23 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0052] The display device 24 is a device that displays images and text information. The display device 24 displays various images under the control of the processing device 21. For example, various display panels such as a liquid crystal panel and an organic EL (Electro Luminescence) panel are suitably used as the display device 24.

[0053] The input device 25 receives operations from the user U. For example, the input device 25 includes a keyboard, a touchpad, a touch panel, and a pointing device such as a mouse. Here, if the input device 25 includes a touch panel, it may also serve as the display device 24.

[0054] The positioning device 26 acquires position information of the terminal device 20. The positioning device 26 may be, for example, a GNSS (Global Navigation Satellite System) receiver. The GNSS receiver receives radio signals transmitted from one or more GNSS satellites. GNSS is a positioning system that uses positioning satellites from countries around the world, including GPS (Global Positioning System) satellites. The radio signals include information such as the position information of the satellite that transmitted the radio signals and the transmission time of the radio signals. The GNSS receiver performs positioning based on the one or more received radio signals, and outputs the position information of the GNSS receiver to the processing device 21. The position information of the GNSS receiver indicates, for example, the latitude and longitude of the terminal device 20.

[0055] The processing device 21 functions as an acquisition unit 211, a determination unit 212, a determination unit 213, and a display control unit 214, for example, by reading and executing a control program PR2 from the storage device 22.

[0056] The acquisition unit 211 acquires direction information and line-of-sight information. Here, the direction information will be explained first. The direction information is information regarding the direction in which the user U should move. Hereinafter, the direction in which the user U should move is also referred to as the target traveling direction Df. The acquisition unit 211 acquires imaging information from the imaging device 35 of the MR glasses 30 via the communication device 23. The acquisition unit 211 grasps the situation in which the user U is placed based on the acquired imaging information. The imaging information includes image information about the external area in front of the user U. The acquisition unit 211 recognizes the direction of the road and the traveling direction of surrounding pedestrians from the image information using well-known image recognition technology.

[0057] Fig. 7 is a diagram showing an example of an image AG1 captured by the imaging device 35 while the user U is walking. Fig. 8 is a diagram showing an example of an image AG2 captured a certain time after the image shown in Fig. 7 was captured. Images AG1 and AG2 show the sidewalk SW, curbstone SC, person P1, person P2, etc.

[0058] The acquisition unit 211 recognizes the curbstone SC as an object that indicates the direction in which the sidewalk SW extends. As a result, the acquisition unit 211 recognizes the longitudinal direction of the curbstone SC as the direction in which the user U should move. Furthermore, the acquisition unit 211 can identify the moving direction of the person P1 from the difference between the position of the person P1 in the image AG1 and the position of the person P1 in the image AG2. The acquisition unit 211 recognizes that the direction in which the user U should move is the opposite direction to the moving direction of the person P1. Furthermore, the acquisition unit 211 recognizes the roadside tree T1 as a stationary object. The acquisition unit 211 recognizes that the position of the roadside tree T1 in the image AG1 is different from the position of the roadside tree T1 in the image AG2, and therefore recognizes that the user U is moving.

[0059] In this way, the acquisition unit 211 acquires direction information based on image information. This direction information is the direction in which the imaging device 35 is facing, that is, the direction relative to the direction in which the face of the user U is facing.

[0060] In addition, the acquisition unit 211 may identify the position of the user U based on the position information of the terminal device 20 obtained by the positioning device 26, and determine the direction in which the user U should proceed based on the identified position of the user U and the map information stored in the storage device 22.

[0061] Next, the gaze information will be described. The gaze information is information relating to the gaze of the user U. The acquisition unit 211 acquires the movement information from the movement detection device 34 via the communication device 23, and also acquires the gaze direction information from the gaze acquisition device 33. The acquisition unit 211 determines the gaze information based on the movement information and the gaze direction information.

[0062] The determination unit 212 determines the line-of-sight direction Dv based on the line-of-sight information. The determination unit 212 determines the target traveling direction Df based on the direction information. This will be described in detail below with reference to FIG.

[0063] 9 is a diagram showing the positional relationship between the MR glasses 30 and the first virtual object VO1. As shown in FIG. 9, the user U is gazing at the first virtual object VO1. The first virtual object VO1 is placed on the plane SS of the celestial sphere CS, as described with reference to FIG. 2. Furthermore, in this example, the first virtual object VO1 is placed on the equator EQ.

[0064] The direction in which the MR glasses 30 are facing, i.e., the direction Dc in which the imaging device 35 is facing, is set as the reference direction. If the angle formed between the line of sight Dv of the user U and the direction Dc in which the imaging device 35 is facing is set as a first angle θ1, the first angle θ1 can be obtained from the line of sight direction information acquired by the line of sight acquisition device 33. The line of sight direction Dv is the direction in which the virtual object VO that the user U is gazing at is viewed from the user U. The line of sight direction Dv is an example of the first direction.

[0065] If the angle between the target traveling direction Df and the direction Dc in which the imaging device 35 is facing is defined as a second angle θ2, the second angle θ2 can be obtained from an image captured by the imaging device 35. The target traveling direction Df is the direction in which the user U should travel. The target traveling direction Df is an example of a second direction. The angle θd between the target traveling direction Df and the line of sight direction Dv is expressed as θ1 + θ2.

[0066] 10 is a schematic diagram showing an example of moving the first virtual object VO1. The determination unit 213 first determines whether the user U is moving based on the captured images included in the direction information. For example, if the same stationary object captured in two captured images captured at different times is recognized to be in different positions, the determination unit 213 determines that the user U is moving.

[0067] The determination of whether the user U is moving is not limited to being made based on the captured image. For example, the determination of whether the user U is moving may be made based on a change in position information obtained by the positioning device 26, or may be made based on information related to vibrations detected by the motion detection device 34.

[0068] If it is determined that the user U is moving, the determination unit 213 determines whether the line of sight direction Dv and the target traveling direction Df differ from each other. More specifically, the determination unit 213 determines whether the angle θd formed between the line of sight direction Dv and the target traveling direction Df exceeds a threshold angle θth.

[0069] The threshold angle θth is set to, for example, 20 degrees. That is, when the angle formed between the line of sight direction Dv and the target traveling direction Df exceeds 20 degrees, the determination unit 213 determines that the line of sight direction Dv and the target traveling direction Df are different from each other. Here, 20 degrees is merely an example, and the threshold angle θth is not limited to 20 degrees.

[0070] 10, the angle θd1 between the line of sight Dv and the target traveling direction Df with respect to the source first virtual object VO1 is greater than the threshold angle θth. In this case, the display control unit 214 moves the first virtual object VO1 to a position where the angle θd between the line of sight Dv and the target traveling direction Df becomes smaller. In other words, when the line of sight Dv and the target traveling direction Df differ from each other, the display control unit 214 moves the first virtual object VO1 to a position where the angle θd between the line of sight Dv and the target traveling direction Df becomes smaller. Therefore, in FIG. 10, the angle θd2 between the line of sight Dve and the target traveling direction Df with respect to the destination first virtual object VO1e is smaller than the threshold angle θth.

[0071] 11 is a schematic diagram showing another example of moving the first virtual object VO1. As shown in FIG. 11, the angle θd2 between the line of sight direction Dve with respect to the destination first virtual object VO1e and the target traveling direction Df is 0 degrees. According to this aspect, the first virtual object VO1 is placed in a position that is most visible to the user U. As a result, the direction in which the user U gazes and the direction in which the user U should move coincide with each other, so the user U can easily move in the direction in which the user U should move.

[0072] 11, the user U sees the scenery in the direction in which the user U should travel through the first virtual object VO1. In contrast, in the example shown in Fig. 10, the target traveling direction Df and the line of sight direction Dve are different from each other, so the scenery in the target traveling direction Df is not obstructed by the first virtual object VO1. This allows the user U to easily view the scenery in the target traveling direction Df while gazing at the first virtual object VO1.

[0073] The destination position of the first virtual object VO1 may be set in advance by the user U, or may be set according to the situation around the user U, the situation of movement of the user U, etc. For example, the degree of congestion around the user U may be evaluated from imaging information captured by the imaging device 35, and the destination position of the first virtual object VO1 may be changed according to the degree of congestion.

[0074] More specifically, when the degree of congestion around the user U is lower than a predetermined degree, the position of the destination first virtual object VO1e may be set so that the angle θd2 is 0 degrees, as in the example shown in Fig. 11. When the degree of congestion around the user U is equal to or higher than a predetermined degree, the position of the destination first virtual object VO1e may be set so that the angle θd2 is greater than 0 degrees and equal to or smaller than the threshold angle θth, as in the example shown in Fig. 10. Alternatively, the position of the destination first virtual object VO1e may be set so that the angle θd2 becomes smaller as the degree of congestion around the user U becomes lower.

[0075] In addition, the moving speed of the user U may be calculated based on at least one of the imaging information from the imaging device 35 and the position information from the positioning device 26, and the destination position of the first virtual object VO1 may be changed according to the moving speed of the user U.

[0076] More specifically, when the moving speed of the user U is slower than a predetermined speed, the position of the destination first virtual object VO1e may be set so that the angle θd2 is 0 degrees, as in the example shown in Fig. 11. When the moving speed of the user U is equal to or greater than a predetermined speed, the position of the destination first virtual object VO1e may be set so that the angle θd2 is greater than 0 degrees and equal to or less than the threshold angle θth, as in the example shown in Fig. 10. Alternatively, the position of the destination first virtual object VO1e may be set so that the angle θd2 becomes smaller as the moving speed of the user U becomes slower.

[0077] 1.2. Operation of the display control device according to the first embodiment 1.2.1. Operation of the Processing Device 21 Fig. 12 is a flowchart showing the operation of the processing device 21 of Fig. 4. Hereinafter, the operation of the processing device 21 will be described with reference to Fig. 12. The routine of Fig. 12 is started, for example, when the processing device 21 is started, and is executed every time a certain time period has elapsed.

[0078] In step S11, the processing device 21 functions as the acquisition unit 211 to acquire line-of-sight information from the line-of-sight acquisition device 33 and the movement detection device 34, and acquires direction information from the imaging device 35.

[0079] In step S12, the processing device 21 determines whether or not the user U is moving by functioning as the determination unit 213. Whether or not the user U is moving is determined based on the captured image included in the direction information.

[0080] If it is determined that the user U is not moving, that is, if the determination result in step S12 is negative, the processing device 21 temporarily ends this routine. That is, if the user U is not moving, the position where the first virtual object VO1 is placed will not change even if the user U is gazing at the first virtual object VO1.

[0081] On the other hand, if it is determined that the user U is moving, i.e., if the determination result in step S12 is positive, the processing device 21 functions as a determination unit 212 and determines the gaze direction Dv based on the acquired gaze information in step S13.

[0082] In step S14, the processing device 21 functions as the determination unit 213 to determine whether or not the virtual object VO is arranged in the line of sight direction Dv.

[0083] If the virtual object VO is not located in the line of sight direction Dv, that is, if the determination result in step S14 is negative, the processing device 21 temporarily ends this routine. That is, in this case, since the virtual object VO to be moved does not exist, the processing to move the virtual object VO is not executed.

[0084] On the other hand, if the virtual object VO is placed in the line of sight direction Dv, that is, if the judgment result in step S14 is positive, the processing device 21 functions as the determination unit 212, and determines the virtual object VO to be moved in step S15.

[0085] In step S16, the processing device 21 functions as the determination unit 212 to determine the target traveling direction Df based on the acquired direction information.

[0086] In step S17, the processing device 21 functions as the determination unit 213 to determine whether the angle θd formed between the line of sight direction Dv and the target traveling direction Df is greater than the threshold angle θth.

[0087] If the angle θd between the line of sight direction Dv and the target traveling direction Df is equal to or smaller than the threshold angle θth, that is, if the determination result in step S17 is negative, the processing device 21 temporarily ends this routine. That is, in this case, since the virtual object VO is located near the target traveling direction Df, it is determined that there is no need to move the virtual object VO.

[0088] On the other hand, if the angle θd between the line of sight direction Dv and the target traveling direction Df is greater than the threshold angle θth, that is, if the judgment result in step S17 is positive, the processing device 21 functions as the display control unit 214, and in step S18, moves the virtual object VO to be moved so that the angle θd between the line of sight direction Dv and the target traveling direction Df is equal to or less than the threshold angle θth, and temporarily terminates this routine.

[0089] 1.3. Advantages of the First Embodiment According to the above description, the terminal device 20 according to the first embodiment includes the determination unit 212 and the display control unit 214. The determination unit 212 determines a line of sight direction Dv and a target traveling direction Df. The line of sight direction Dv is the direction in which the user U, wearing the MR glasses 30, views a virtual object VO that the user U is gazing at. The target traveling direction Df is the direction in which the user U should travel. When the angle θd formed between the line of sight direction Dv and the target traveling direction Df is larger than the threshold angle θth, the display control unit 214 moves the virtual object VO to a position where the angle θd formed between the line of sight direction Dv and the target traveling direction Df becomes smaller.

[0090] In the MR glasses 30, if the virtual object VO that the user U is gazing at is displayed in a direction different from the direction in which the user U should move, the user U may walk in a direction that deviates from the direction in which the user U should move. According to this aspect, when the angle θd between the line of sight direction Dv and the target moving direction Df is larger than the threshold angle θth, the line of sight direction Dv approaches the target moving direction Df, thereby preventing the actual moving direction of the user U from deviating from the direction in which the user U should move.

[0091] Moreover, the terminal device 20 according to the first embodiment includes an acquisition unit 211. The acquisition unit 211 acquires gaze information and direction information. The gaze information is information relating to the gaze of the user U. The direction information is information relating to the direction in which the user U should move. The determination unit 212 determines a gaze direction Dv based on the gaze information, and determines a target traveling direction Df based on the direction information.

[0092] According to this aspect, the line of sight direction Dv can be determined accurately because the line of sight direction Dv is determined based on actually acquired information about the line of sight of the user U. Also, the target traveling direction Df can be determined accurately because the target traveling direction Df is determined based on actually acquired information about the direction in which the user U should move.

[0093] The MR glasses 30 also include a line-of-sight acquisition device 33, a motion detection device 34, and an imaging device. Line-of-sight information is acquired by the line-of-sight acquisition device 33 and the motion detection device 34. Direction information is acquired by the motion detection device 34.

[0094] According to this aspect, the line of sight information and direction information are acquired by devices provided in the MR glasses 30, so that the line of sight direction Dv and the target traveling direction Df are determined more accurately.

[0095] Furthermore, when the angle θd between the line of sight direction Dv and the target traveling direction Df is greater than the threshold angle θth, the display control unit 214 moves the first virtual object VO1 to a position where the line of sight direction Dv and the target traveling direction Df coincide with each other.

[0096] According to this aspect, the first virtual object VO1 is placed at a position that is most easily visible to the user U. This causes the direction in which the user U gazes and the direction in which the user U should move to coincide with each other, allowing the user U to easily move in the direction in which the user U should move.

[0097] In addition, when the angle θd between the line of sight direction Dv and the target traveling direction Df is greater than the threshold angle θth, the display control unit 214 moves the first virtual object VO1 to a position where the angle θd between the line of sight direction Dv and the target traveling direction Df is greater than 0 degrees and less than or equal to the threshold angle θth.

[0098] According to this aspect, the angle θd between the target traveling direction Df and the line of sight direction Dv is greater than the threshold angle θth, so the first virtual object VO1 does not obstruct the scenery in the target traveling direction Df, allowing the user U to easily view the scenery in the target traveling direction Df while gazing at the first virtual object VO1.

[0099] Furthermore, the display control method according to the first embodiment determines a line of sight Dv and a target traveling direction Df, and if the angle θd between the line of sight Dv and the target traveling direction Df is greater than a threshold angle θth, moves the virtual object VO to a position where the angle θd between the line of sight Dv and the target traveling direction Df becomes smaller. The line of sight Dv is the direction in which the user U, wearing the MR glasses 30, views the virtual object VO that the user U is gazing at. The target traveling direction Df is the direction in which the user U should travel.

[0100] In the MR glasses 30, if the virtual object VO that the user U is gazing at is displayed in a direction different from the direction in which the user U should move, the user U may walk in a direction that deviates from the direction in which the user U should move. According to this aspect, when the angle θd between the line of sight direction Dv and the target moving direction Df is larger than the threshold angle θth, the line of sight direction Dv approaches the target moving direction Df, thereby preventing the actual moving direction of the user U from deviating from the direction in which the user U should move.

[0101] 2. Variations The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples may be combined. Furthermore, the above-described exemplary embodiments and the following modified embodiments may be combined in any combination as long as they are not mutually inconsistent.

[0102] 2.1. Variation 1 When moving the first virtual object VO1 that the user U is gazing at, the first virtual object VO1 may be moved in stages to a final position, i.e., a target position. Fig. 13 is a schematic diagram showing an example of the operation of the processing device 21 of the terminal device 20 according to Modification 1. The display control unit 214 sets the target position to a position where the angle θd between the line of sight direction Dv and the target traveling direction Df is equal to or smaller than a threshold angle θth. The display control unit 214 moves the first virtual object VO1 in stages to the target position toward a straight line L1 along the target traveling direction Df.

[0103] More specifically, the display control unit 214 moves the first virtual object VO1 toward the line L1 by a predetermined angle at regular intervals. That is, the display control unit 214 first moves the first virtual object VO1 to the position indicated by the first virtual object VO1a after a certain period of time has elapsed. After another certain period of time has elapsed, the display control unit 214 moves the first virtual object VO1 to the position indicated by the first virtual object VO1b. In this way, the display control unit 214 moves the first virtual object VO1 to the position indicated by the first virtual object VO1c, the position indicated by the first virtual object VO1d, and the position indicated by the first virtual object VO1e each time a certain period of time has elapsed.

[0104] In this way, the display control unit 214 moves the first virtual object VO1 in stages toward the straight line L1 along the target traveling direction Df.

[0105] According to this aspect, the line of sight of the user U can easily follow the movement of the first virtual object VO1, so the first virtual object VO1 can be moved without giving the user U a sense of discomfort.

[0106] 2.2. Variation 2 14 is a schematic diagram showing another example of moving the first virtual object VO1. When the shape of the first virtual object VO1 is planar, the display control unit 214 may orient the normal NL1 of the first virtual object VO1 parallel to the target traveling direction Df. Examples of planar virtual objects include applications that display still images, videos, and the like, browsers, and applications for generating document files.

[0107] 14, the target traveling direction Df and the normal line NL1 of the first virtual object VO1 coincide with each other. According to this aspect, the user U can easily imagine a virtual line parallel to the normal line NL1 of the first virtual object VO1, and therefore the user U can easily move along the target traveling direction Df.

[0108] 2.3. Variation 3 15 is a schematic diagram showing another example of moving the first virtual object VO1. When the angle θd between the line of sight Dv and the target traveling direction Df is greater than the threshold angle θth, the display control unit 214 may move the first virtual object VO1 to a position beyond the line L1 as seen from the first virtual object VO1 and to a position where the angle θd between the line of sight Dv and the target traveling direction Df is equal to or smaller than the threshold angle θth.

[0109] 15, the destination first virtual object VO1f is disposed on either side of the source first virtual object VO1 by a straight line L1. For example, when a virtual object other than the first virtual object VO1 is already disposed between the source first virtual object VO1 and the straight line L1, the display control according to this modification is effective.

[0110] That is, even if there is a virtual object other than the first virtual object VO1 between the source first virtual object VO1 and the line L1, the first virtual object VO1 can be placed within a range equal to or smaller than the threshold angle θth. Therefore, according to this embodiment, the degree of freedom in placing the first virtual object VO1 is improved.

[0111] 2.4. Variation 4 In the first embodiment, the terminal device 20 functions as a display control device, but the MR glasses 30 may function as a display control device instead of the terminal device 20, or a control server (not shown) may function as a display control device. The control server is connected to the server 10 and the MR glasses 30 via a communication network NET so as to be able to communicate with each other.

[0112] 2.5. Variation 5 In the first embodiment, a wearable display device employing an optical see-through system has been described as an example of the MR glasses 30, but the MR glasses 30 may employ a wearable display device employing a video see-through system.

[0113] 3.Other (1) In the above-described embodiment, storage device 12, storage device 22, and storage device 32 are exemplified by ROM and RAM, but may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0114] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0115] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0116] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0117] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0118] (6) Each function illustrated in Figures 1 to 15 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically coupled, or may be realized by using two or more devices that are physically or logically separated and connected directly or indirectly (for example, by wire, wirelessly, etc.). A functional block may also be realized by combining software with the one device or the multiple devices.

[0119] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0120] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0121] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0122] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0123] (10) In the above-described embodiments, the server 10 may be a mobile station (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate terminology. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0124] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0125] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0126] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0127] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0128] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include the nouns following these articles being plural.

[0129] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0130] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0131] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0132] 1...information processing system, 20...terminal device, 30...MR glasses, 33...gaze acquisition device, 34...motion detection device, 35...imaging device, 211...acquisition unit, 212...determination unit, 214...display control unit, Df...target direction of travel, Dv, Dve...gaze direction, L1...straight line, NL1...normal, U...user, VO, VO1, VO2, VO3, VO4, VO5...virtual object, VS...virtual space, θd...angle, θth...threshold angle.

Claims

1. a determination unit that determines a first direction in which a virtual object that a user wearing a see-through wearable device gazes at is viewed from the user, and a second direction in which the user should move; a display control unit that moves the virtual object to a position where the angle formed between the first direction and the second direction becomes smaller when the angle formed between the first direction and the second direction is larger than a threshold angle; A display control device comprising:

2. an acquisition unit that acquires line-of-sight information regarding the line of sight of the user and direction information regarding a direction in which the user should move; the determination unit determines the first direction based on the line-of-sight information and determines the second direction based on the direction information. The display control device according to claim 1 .

3. the see-through wearable device includes a line-of-sight acquisition device, a motion detection device, and an imaging device; the line-of-sight information is acquired by the line-of-sight acquisition device and the movement detection device, the direction information is acquired by the imaging device; The display control device according to claim 2 .

4. the display control unit moves the virtual object in a stepwise manner toward a straight line along the second direction; The display control device according to claim 1 .

5. When the shape of the virtual object is planar, the display control unit makes the direction of a normal of the virtual object parallel to the second direction. The display control device according to claim 1 .

6. When the angle between the first direction and the second direction is greater than the threshold angle, the display control unit moves the virtual object to a position where the first direction and the second direction are aligned. The display control device according to claim 1 .

7. when the angle formed between the first direction and the second direction is greater than the threshold angle, the display control unit moves the virtual object to a position where the angle formed between the first direction and the second direction is greater than 0 degrees and equal to or less than the threshold angle. The display control device according to claim 1 .

8. determining a first direction in which a virtual object gazed at by a user wearing a see-through wearable device is viewed from the user, and a second direction in which the user should move; When the angle between the first direction and the second direction is larger than a threshold angle, the virtual object is moved to a position where the angle between the first direction and the second direction becomes smaller. Display control method.

Images