Information processing device

Abstract

This information processing device comprises: an acquisition unit that acquires visual line information indicating the visual line of a user; a display control unit that displays a first virtual object and a second virtual object at positions not overlapping each other, as seen from the user, in a virtual space; and an operation control unit that causes the second virtual object to transition from inactive to active after a transition has occurred from a first state where the visual line of the user indicated by the visual line information intersects the first virtual object and the first virtual object is active to a second state where the visual line of the user does not intersect the first virtual object.

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In MR (Mixed Reality) technology, the real environment perceived by the user is augmented by a computer. Using this technology, it is possible, for example, to precisely overlay a virtual space on the real space perceived through MR glasses worn by the user.

[0003] For example, Patent Document 1 discloses an information processing device that displays realistic virtual reality images while ensuring the safety of a user riding in a vehicle. The information processing device includes AR goggles that display images of multiple virtual first objects. The information processing device also detects a second object that actually exists outside or inside the vehicle. If the information processing device determines that the image of the first object will be superimposed on a visibility ensuring area that affects the user's visibility of the second object, it moves the position of the image of the first object, for example, so that the image of the first object does not superimpose on the visibility ensuring area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-204984 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 defines the positional relationship between a virtual first object and a real second object, and is not able to change the state of the second object depending on which object the user is focusing on.

[0006] Therefore, the present invention aims to solve the problem of changing the state of one of two virtual objects placed in a virtual space when the user's focus changes from a state in which the user is focusing on the other virtual object. [Means for solving the problem]

[0007] An information processing device according to a preferred embodiment of the present invention is an information processing device including: an acquisition unit that acquires gaze information indicating a user's gaze; a display control unit that displays a first virtual object and a second virtual object at positions in a virtual space where they do not overlap each other as seen from the user; and an operation control unit that transitions the second virtual object from inactive to active after a first state in which the user's gaze indicated by the gaze information intersects with the first virtual object and the first virtual object is active transitions to a second state in which the user's gaze does not intersect with the first virtual object. [Effects of the Invention]

[0008] According to the present invention, when a user focuses on one of two virtual objects placed in a virtual space and there is a change in the object the user focuses on, it is possible to change the state of the other virtual object. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment. [Figure 2] 1 is a perspective view showing the appearance of MR glasses 20 according to a first embodiment. [Figure 3] 1 is a schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 20 according to the first embodiment. [Figure 4] 1 is a schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 20 according to the first embodiment. [Figure 5]FIG. 2 is a block diagram showing an example of the configuration of MR glasses 20 according to the first embodiment. [Figure 6] FIG. 1 is a block diagram showing an example of the configuration of a terminal device 10 according to a first embodiment. [Figure 7] FIG. 3 is an explanatory diagram illustrating an example of the operation of the display control unit 112 according to the first embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of a server 30 according to the first embodiment. [Figure 9] 5 is a flowchart showing the operation of the server 30 according to the first embodiment. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of the operation of a display control unit 112 according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1: First embodiment Hereinafter, with reference to FIGS. 1 to 9, a description will be given of the configuration of an information processing system 1 including a terminal device 10 as an information processing device according to a first embodiment of the present invention.

[0011] 1-1: Configuration of the first embodiment 1-1-1: Overall structure 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment of the present invention. The information processing system 1 is a system that provides a virtual space to a user U1 wearing MR glasses 20 (described later) using MR technology.

[0012] The information processing system 1 includes a terminal device 10, MR glasses 20, and a server 30. The terminal device 10 is an example of an information processing device. In the information processing system 1, the terminal device 10 and the server 30 are connected to each other so as to be able to communicate with each other via a communication network NET. Furthermore, the terminal device 10 and the MR glasses 20 are connected to each other so as to be able to communicate with each other. Note that in FIG. 1, three pairs of terminal devices 10 and MR glasses 20 are shown, namely, a pair of terminal device 10-1 and MR glasses 20-1, a pair of terminal device 10-2 and MR glasses 20-2, and a pair of terminal device 10-3 and MR glasses 20-3. However, this number of pairs is merely an example, and the information processing system 1 can include any number of pairs of terminal devices 10 and MR glasses 20.

[0013] The server 30 provides various data and cloud services to the terminal device 10 via the communication network NET.

[0014] The terminal device 10 displays virtual objects arranged in a virtual space on the MR glasses 20 worn by the user on the head. The virtual space is, for example, a celestial sphere. Examples of the virtual objects include virtual objects representing data such as still images, videos, 3DCG models, HTML files, and text files, as well as virtual objects representing applications. Examples of text files include memos, source code, diaries, and recipes. Examples of applications include browsers, applications for using SNS, and applications for generating document files. The terminal device 10 is preferably, for example, a mobile terminal device such as a smartphone or a tablet.

[0015] The MR glasses 20 are a see-through wearable display worn on the user's head. The MR glasses 20 display virtual objects on display panels provided in each of the lenses for both eyes under the control of the terminal device 10. The MR glasses 20 are an example of a display device.

[0016] 1-1-2: Composition of MR glasses Fig. 2 is a perspective view showing the appearance of the MR glasses 20. As shown in Fig. 2, the appearance of the MR glasses 20 is similar to that of ordinary eyeglasses, and includes temples 91 and 92, a bridge 93, frames 94 and 95, and lenses 41L and 41R. An imaging device 26 is provided on the bridge 93. The imaging device 26 captures an image of the outside world. The imaging device 26 also outputs imaging information indicating the captured image.

[0017] Each of the lenses 41L and 41R is provided with a half mirror. The frame 94 is provided with a liquid crystal panel or organic EL panel for the left eye and an optical member that guides light emitted from the display panel for the left eye to the lens 41L. The liquid crystal panel or organic EL panel will hereinafter be collectively referred to as the display panel. The half mirror provided on the lens 41L transmits external light to guide it to the left eye and reflects light guided by the optical member to make it incident on the left eye. The frame 95 is provided with a display panel for the right eye and an optical member that guides light emitted from the display panel for the right eye to the lens 41R. The half mirror provided on the lens 41R transmits external light to guide it to the right eye and reflects light guided by the optical member to make it incident on the right eye.

[0018] The display 28, which will be described later, includes a lens 41L, a display panel for the left eye, and an optical member for the left eye, as well as a lens 41R, a display panel for the right eye, and an optical member for the right eye.

[0019] In the above configuration, the user can observe the image displayed on the display panel in a see-through state, superimposed on the outside world. In addition, by displaying the image for the left eye on the display panel for the left eye and the image for the right eye on the display panel for the right eye, among the binocular images with parallax in the MR glasses 20, the user U1 can perceive the displayed image as if it has depth and a three-dimensional effect.

[0020] 3 and 4 are schematic diagrams of a virtual space VS provided to a user U1 by using the MR glasses 20. As shown in FIG. 3, virtual objects VO1 to VO5 representing various contents such as a browser, cloud services, images, and videos are arranged in the virtual space VS. The user U1 can experience the virtual space VS as a private space in a public space by moving around the public space while wearing the MR glasses 20 displaying the virtual objects VO1 to VO5 arranged in the virtual space VS. Consequently, the user U1 can act in the public space while receiving the benefits provided by the virtual objects VO1 to VO5 arranged in the virtual space VS.

[0021] 4, it is also possible for multiple users U1 to U3 to share a virtual space VS. By sharing the virtual space VS among multiple users U1 to U3, the multiple users U1 to U3 can share one or more virtual objects VO, and can communicate with each other via the shared virtual objects VO.

[0022] 3 and 4, among the plurality of virtual objects VO1 to VO5, any combination of two virtual objects VO is displayed so as not to overlap with each other in the virtual space VS as viewed from the user U1. Details of the method of displaying the virtual objects VO will be described later.

[0023] 5 is a block diagram showing an example configuration of the MR glasses 20. The MR glasses 20 include a processing device 21, a storage device 22, a gaze detection device 23, a GPS device 24, a motion detection device 25, an imaging device 26, a communication device 27, a display 28, and a speaker 29. The elements of the MR glasses 20 are connected to each other by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, or unit.

[0024] The processing device 21 is a processor that controls the entire MR glasses 20. The processing device 21 is configured using, for example, one or more chips. The processing device 21 is also configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 21 may be realized by hardware such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). The processing device 21 executes various processes in parallel or sequentially.

[0025] The storage device 22 is a recording medium that can be read from and written to by the processing device 21. The storage device 22 also stores a plurality of programs including a control program PR1 that the processing device 21 executes.

[0026] The gaze detection device 23 detects the gaze of the user U1. Any method may be used for detecting the gaze by the gaze detection device 23. For example, the gaze detection device 23 may detect gaze information based on the position of the inner corner of the eye and the position of the iris. Furthermore, the gaze detection device 23 outputs gaze information indicating the direction of the gaze of the user U1 based on the detection result to the processing device 21 described below. The gaze information output to the processing device 21 is output to the terminal device 10 via the communication device 27.

[0027] The GPS device 24 receives radio waves from multiple satellites. The GPS device 24 also generates location information from the received radio waves. The location information indicates the location of the MR glasses 20. The location information may be in any format as long as it can identify the location. The location information indicates, for example, the latitude and longitude of the MR glasses 20. As an example, the location information is obtained from the GPS device 24. However, the MR glasses 20 may obtain the location information by any method. The obtained location information is supplied to the processing device 21. The location information output to the processing device 21 is transmitted to the terminal device 10 via the communication device 27.

[0028] The motion detection device 25 detects the motion of the MR glasses 20. Examples of the motion detection device 25 include inertial sensors such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration. The acceleration sensor detects acceleration on orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes as the central axes of rotation. The motion detection device 25 can generate attitude information indicating the attitude of the MR glasses 20 based on the output information of the gyro sensor. The motion detection device 25 supplies attitude information related to the attitude of the MR glasses 20 to the processing device 21. The motion detection device 25 also supplies motion information related to the motion of the MR glasses 20 to the processing device 21. The motion information includes acceleration data indicating the acceleration on each of the three axes and angular acceleration data indicating the angular acceleration on each of the three axes. The attitude information and motion information supplied to the processing device 21 are transmitted to the terminal device 10 via the communication device 27.

[0029] The imaging device 26 outputs imaging information obtained by capturing an image of the outside world. The imaging device 26 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is supplied to the processing device 21. The imaging information supplied to the processing device 21 is transmitted to the terminal device 10 via the communication device 27.

[0030] The communication device 27 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 27 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 27 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 27 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0031] The display 28 is a device that displays images. The display 28 displays various images under the control of the processing device 21. As described above, the display 28 includes the lens 41L, a display panel for the left eye, and an optical member for the left eye, as well as the lens 41R, a display panel for the right eye, and an optical member for the right eye. As the display panel, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used.

[0032] The speaker 29 is a device that emits sound. The speaker 29 emits various sounds under the control of the processing device 21. For example, sound data, which is a digital signal, is converted into a sound signal, which is an analog signal, by a DA converter (not shown). The amplitude of the sound signal is amplified by an amplifier (not shown). The speaker 29 emits sound represented by the sound signal after the amplitude has been amplified.

[0033] The processing device 21 functions as an acquisition unit 211 and a display control unit 212, for example, by reading out a control program PR1 from the storage device 22 and executing it.

[0034] The acquisition unit 211 acquires image information indicating an image to be displayed on the MR glasses 20 from the terminal device 10. More specifically, the acquisition unit 211 acquires, for example, second image information (described later) transmitted from the terminal device 10.

[0035] Furthermore, the acquisition unit 211 acquires gaze information input from the gaze detection device 23, position information input from the GPS device 24, posture information and movement information input from the movement detection device 25, and imaging information input from the imaging device 26. Then, the acquisition unit 211 outputs the acquired gaze information, position information, posture information, movement information, and imaging information to the communication device 27.

[0036] Based on the second image information acquired by the acquisition unit 211 from the terminal device 10, the display control unit 212 causes the display 28 to display an image indicated by the second image information.

[0037] 1-1-3: Terminal device configuration 6 is a block diagram showing an example configuration of the terminal device 10. The terminal device 10 includes a processing device 11, a storage device 12, a communication device 13, a display 14, an input device 15, and an inertial sensor 16. The elements of the terminal device 10 are connected to each other by one or more buses for communicating information.

[0038] The processing device 11 is a processor that controls the entire terminal device 10. The processing device 11 is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 11 executes various processes in parallel or sequentially.

[0039] The storage device 12 is a recording medium that can be read from and written to by the processing device 11. The storage device 12 also stores a plurality of programs including a control program PR2 that the processing device 11 executes.

[0040] The communication device 13 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0041] The display 14 is a device that displays images and text information. The display 14 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 14.

[0042] The input device 15 receives operations from a user U1 wearing the MR glasses 20 on their head. For example, the input device 15 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 15 includes a touch panel, it may also serve as the display 14.

[0043] The inertial sensor 16 is a sensor that detects inertial force. The inertial sensor 16 includes, for example, one or more of an acceleration sensor, an angular velocity sensor, and a gyro sensor. The processing device 11 detects the attitude of the terminal device 10 based on the output information of the inertial sensor 16. Furthermore, the processing device 11 accepts the selection of a virtual object VO, the input of text, and the input of instructions in the spherical virtual space VS based on the attitude of the terminal device 10. For example, the user U1 operates the input device 15 while pointing the central axis of the terminal device 10 toward a predetermined area in the virtual space VS, thereby selecting a virtual object VO to be placed in the predetermined area. The operation by the user U1 on the input device 15 is, for example, a double tap. By operating the terminal device 10 in this way, the user U1 can select a virtual object VO without looking at the input device 15 of the terminal device 10.

[0044] The processing device 11 functions as an acquisition unit 111, a display control unit 112, a determination unit 113, and an operation control unit 114 by reading and executing the control program PR2 from the storage device 12.

[0045] The acquisition unit 111 acquires line-of-sight information indicating the line of sight of the user U1. More specifically, the acquisition unit 111 acquires line-of-sight information output from the MR glasses 20 via the communication device 13. In addition, the acquisition unit 111 acquires first image information (described later) indicating an image displayed on the MR glasses 20 from the server 30 via the communication device 13.

[0046] The display control unit 112 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as seen by the user U1, based on the first image information acquired by the acquisition unit 111. In detail, the first image information is image information showing images of the first virtual object VO1 itself and the second virtual object VO2 itself. In addition, the display control unit 112 transmits second image information including the above-mentioned first image information and layout information such that the first virtual object VO1 and the second virtual object VO2 do not overlap each other as seen by the user U1 in the virtual space VS viewed by the user U1 through the MR glasses 20, to the MR glasses 20 via the communication device 13.

[0047] FIG. 7 is an explanatory diagram of a display method of a first virtual object VO1 and a second virtual object VO2 by the display control unit 112. In the following description, assume that the virtual space VS has an X-axis, a Y-axis, and a Z-axis that are orthogonal to each other. As an example, the X-axis extends left and right from the user U1. Furthermore, from the user U1's perspective, the right direction along the X-axis is the positive direction, and the left direction along the X-axis is the negative direction. Furthermore, the Y-axis extends forward and backward from the user U1's perspective. Furthermore, from the user U1's perspective, the forward direction along the Y-axis is the positive direction, and the backward direction along the Y-axis is the negative direction. These X-axis and Y-axis form a horizontal plane. Furthermore, the Z-axis is orthogonal to the XY plane and extends up and down from the user U1. Furthermore, from the user U1's perspective, the upward direction along the Z-axis is the positive direction, and the downward direction along the Z-axis is the negative direction.

[0048] In the virtual space VS, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1, i.e., the coordinates of one end point of the line of sight V of the user U1, is assumed to be (x, y, z) = (x0, y0, z0). Furthermore, the display control unit 112 displays the first virtual object VO1 at the position (x, y, z) = (x1, y1, z1).

[0049] Here, a cylinder Y is assumed as an area where the line of sight V of the user U1 may exist while the user U1 is viewing the first virtual object VO1. The cylinder Y has a central axis L1 with a first endpoint C1 at the position (x0, y0, z0) of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1 and a second endpoint C2 at the position (x1, y1, z1) of the first virtual object VO1. Furthermore, the radius R1 of the bottom of the cylinder Y is a length that allows the bottom to include the first virtual object VO1. Note that, for convenience of explanation, in FIG. 7, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1 does not coincide with the position of the first endpoint C1. Similarly, in FIG. 7, the position (x1, y1, z1) of the first virtual object VO1 does not coincide with the position of the second endpoint C2. However, in practical use of the present invention, the respective positions in these pairs coincide.

[0050] The display control unit 112 displays the second virtual object VO2 in an inactive state outside the cylinder Y in the virtual space VS. Specifically, the display position (x, y, z) = (x2, y2, z2) of the second virtual object VO2 is a position where the entire second virtual object VO2 is outside the range of the cylinder Y. Here, "active" means that the virtual object VO is in operation or has been selected by the user U1. On the other hand, "inactive" means that the operation of the virtual object VO has stopped or has not been selected by the user U1. For example, if the virtual object VO corresponds to a video application, "the virtual object VO is in operation" means that the video is being played by the operation of the user U1. Furthermore, if the virtual object VO corresponds to an email application, "the virtual object VO is in operation" means that the email is being sent or received, or a text file showing the contents of the email is being opened by the operation of the user U1. Furthermore, when the virtual object VO corresponds to a music application, "the virtual object VO is in operation" means that the music is being emitted from the speaker 29 by an operation of the user U1.

[0051] The display control unit 112 displays the second virtual object VO2 in the virtual space VS in a first state in which the line of sight V of the user U1 intersects with the first virtual object VO1 and the first virtual object VO1 is active. Specifically, the display control unit 112 displays the second virtual object VO2 in its entirety outside the cylinder Y in the first state in which the line of sight V of the user U1 is located entirely inside the cylinder Y and the first virtual object VO1 is active.

[0052] The display position of the second virtual object VO2 is an area within a predetermined distance from the first virtual object VO1. Specifically, the second virtual object VO2 is displayed within the field of view of the user U1 while the user U1 is viewing the first virtual object VO1. This process allows the terminal device 10 to draw the user U1's attention to the second virtual object VO2. The above-mentioned "predetermined distance" is an example of the "first distance."

[0053] Returning to the explanation of Figure 6, the determination unit 113 determines whether a first state in which the line of sight V of the user U1 intersects with the first virtual object VO1 and the first virtual object VO1 is active has transitioned to a second state in which the line of sight V of the user U1 does not intersect with the first virtual object VO1. The operation control unit 114 transitions the second virtual object VO2 from inactive to active after the determination unit 113 determines that the first state has transitioned to the second state. Specifically, in Fig. 7, after the line of sight V of the user U1 changes from a state where it fits within the cylinder Y to a state where it does not fit within the cylinder Y, the operation control unit 114 transitions the second virtual object VO2 from inactive to active.

[0054] In the second state, the operation control unit 114 preferably transitions the second virtual object VO2 from inactive to active after the line of sight V of the user U1 remains stationary for a predetermined period of time. Specifically, in the second state, the operation control unit 114 transitions the second virtual object VO2 from inactive to active on the condition that the change per unit time of the line of sight V of the user U1 remains within a predetermined range for a predetermined period of time. As a result, the terminal device 10 can transition the second virtual object VO2 from inactive to active after the line of sight V of the user U1 is no longer wandering and is close to a stationary state. The above-mentioned "predetermined range" is an example of a "second range." Furthermore, the above-mentioned "predetermined time" is an example of a "second time."

[0055] Furthermore, after transitioning from the first state to the second state, the operation control unit 114 transitions the first virtual object VO1 from active to inactive. This process enables the terminal device 10 to switch between the first virtual object VO1 and the second virtual object VO2 to activate, depending on the position of the line of sight V of the user U1. Furthermore, the terminal device 10 can exclusively activate the first virtual object VO1 and the second virtual object VO2. The state transition of the first virtual object VO1 and the state transition of the second virtual object VO2 may be executed simultaneously, or one may be executed before the other.

[0056] 1-1-4: Server configuration 8 is a block diagram showing an example of the configuration of the server 30. The server 30 includes a processing device 31, a storage device 32, a communication device 33, a display 34, and an input device 35. The elements of the server 30 are connected to each other by one or more buses for communicating information.

[0057] The processing device 31 is a processor that controls the entire server 30. The processing device 31 is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 31 may be realized by hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 31 executes various processes in parallel or sequentially.

[0058] The storage device 32 is a recording medium that can be read from and written to by the processing device 31. The storage device 32 also stores a plurality of programs including a control program PR3 that the processing device 31 executes.

[0059] The communication device 33 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0060] The display 34 is a device that displays images and text information. The display 34 displays various images under the control of the processing device 31. For example, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used as the display 34.

[0061] The input device 35 is a device that accepts operations by an administrator of the information processing system 1. For example, the input device 35 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 35 includes a touch panel, it may also serve as the display 34.

[0062] The processing device 31 functions as an acquisition unit 311 and a generation unit 312, for example, by reading out a control program PR3 from the storage device 32 and executing it.

[0063] The acquisition unit 311 acquires various data from the terminal device 10 using the communication device 33. The data includes, for example, data indicating the operation content for the virtual object VO, which is input to the terminal device 10 by a user U1 wearing the MR glasses 20 on his / her head.

[0064] The generation unit 312 generates first image information indicating an image to be displayed on the MR glasses 20. This first image information is transmitted to the terminal device 10 via the communication device 33. As described above, the display control unit 112 provided in the terminal device 10 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as seen by the user U1, based on the first image information received via the communication device 13.

[0065] 1-2: Operation of the first embodiment 9 is a flowchart showing the operation of the terminal device 10 according to the first embodiment. Hereinafter, the operation of the terminal device 10 will be described with reference to FIG.

[0066] In step S1, the processing device 11 functions as the display control unit 112. The processing device 11 displays a first virtual object VO1 in the virtual space VS.

[0067] In step S2, the processing device 11 functions as an acquisition unit 111. The processing device 11 acquires the line of sight V of the user U1.

[0068] In step S3, the processing device 11 functions as the display control unit 112. The processing device 11 displays the second virtual object VO2 in an inactive state in the virtual space VS so that it does not overlap with the first virtual object VO1 as seen by the user U1. Note that the first virtual object VO1 is assumed to be active at the latest in step S3.

[0069] In step S4, the processing device 11 functions as the determination unit 113. The processing device 11 determines whether or not a transition has occurred from a first state in which the line of sight V of the user U1 intersects with the first virtual object VO1 and the first virtual object VO1 is active to a second state in which the line of sight V of the user U1 does not intersect with the first virtual object VO1. If it is determined that a transition has occurred from the first state to the second state, that is, if the determination result of step S4 is positive, the processing device 11 executes the processing of step S5. If it is not determined that a transition has occurred from the first state to the second state, that is, if the determination result of step S4 is negative, the processing device 11 executes the processing of step S4.

[0070] In step S5, the processing device 11 functions as the action control unit 114. The processing device 11 transitions the second virtual object VO2 from inactive to active.

[0071] In step S6, the processing device 11 functions as the operation control unit 114. The processing device 11 transitions the first virtual object VO1 from active to inactive. After that, the processing device 11 ends all the operations shown in FIG.

[0072] 1-3: Effects of the First Embodiment According to the above description, the terminal device 10 as an information processing device includes an acquisition unit 111, a display control unit 112, and an operation control unit 114. The acquisition unit 111 acquires gaze information indicating the gaze V of the user U1. The display control unit 112 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap with each other as seen by the user U1. The operation control unit 114 transitions from a first state in which the gaze V of the user U1 indicated by the gaze information intersects with the first virtual object VO1 and the first virtual object VO1 is active to a second state in which the gaze V of the user U1 does not intersect with the first virtual object VO1, and then transitions the second virtual object VO2 from inactive to active.

[0073] By using the above configuration, the terminal device 10 can newly display the second virtual object VO2 so that it does not overlap with the previously displayed first virtual object VO1. Furthermore, the terminal device 10 can switch which virtual object VO to activate, between the first virtual object VO1 and the second virtual object VO2, depending on the direction of the line of sight V of the user U1.

[0074] Furthermore, according to the above description, after the transition from the first state to the second state, the action control unit 114 transitions the first virtual object VO1 from active to inactive.

[0075] By using the above configuration, the terminal device 10 can exclusively activate the first virtual object VO1 and the second virtual object VO2. For example, if the first virtual object VO1 is a first application and the second virtual object VO2 is a second application, the user U1 can stop the operation of the first application and start the second application by moving his / her line of sight V away from the first virtual object VO1. In other words, the user U1 can switch the application to be operated simply by moving his / her line of sight V away from the first virtual object VO1, without inputting an instruction for the second virtual object VO2.

[0076] Furthermore, according to the above explanation, the display control unit 112 starts displaying the second virtual object VO2 in the virtual space VS in the first state.

[0077] By using the above configuration, the terminal device 10 can make the second virtual object VO2 appear in the first state. For example, the terminal device 10 can display the second application, which is the second virtual object VO2, while the first application, which is the first virtual object VO1, is running.

[0078] Furthermore, according to the above explanation, the operation control unit 114 transitions the second virtual object VO2 from inactive to active, on the condition that the change per unit time of the user U1's line of sight V remains within the second range for a second period of time in the second state.

[0079] By using the above configuration, the terminal device 10 can transition the second virtual object VO2 from inactive to active after the line of sight V of the user U1 is no longer wandering and is close to being stationary.

[0080] Furthermore, according to the above explanation, the display control unit 112 displays the second virtual object VO2 in an area within the first distance from the first virtual object VO1.

[0081] By using the above configuration, the terminal device 10 can increase the degree to which it can attract the user U1's attention to the second virtual object VO2.

[0082] Furthermore, according to the above explanation, the display control unit 112 displays the second virtual object VO2 outside the cylinder Y in the virtual space VS, which has a central axis L1 that coincides with the line of sight V of the user U1 and a bottom surface that includes the first virtual object VO1.

[0083] By using the above configuration, the terminal device 10 can display the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as seen by the user U1.

[0084] 2: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected arbitrarily from the following examples may be combined.

[0085] 2-1: Variation 1 In the terminal device 10 according to the above embodiment, the display control unit 112 displays the second virtual object VO2 in the virtual space VS in the first state. However, the method for displaying the second virtual object VO2 is not limited to this. For example, in the second state, the display control unit 112 may display the second virtual object VO2 at a position intersecting with the line of sight V of the user U1 after the line of sight V of the user U1 remains stationary for a predetermined period of time. In detail, in the second state, the display control unit 112 may display the second virtual object VO2 at a position intersecting with the line of sight V of the user U1 on the condition that the change per unit time of the line of sight V of the user U1 remains within a predetermined range for a predetermined period of time. By displaying the second virtual object VO2 ahead of the line of sight V of the user U1, the user U1 does not need to search for the second virtual object VO2 in the virtual space VS. The "predetermined time" is an example of the "first time." The "predetermined range" is an example of the "first range."

[0086] 2-2: Variation 2 The terminal device 10 according to the above embodiment includes a display control unit 112 and an operation control unit 114. The display control unit 112 displays the first virtual object VO1 and the second virtual object VO2 in positions in the virtual space VS where they do not overlap each other as viewed from the user U1. The operation control unit 114 transitions the second virtual object VO2 from inactive to active after transitioning from a first state in which the line of sight V of the user U1 indicated by the line of sight information intersects with the first virtual object VO1 and the first virtual object VO1 is active to a second state in which the line of sight V of the user U1 does not intersect with the first virtual object VO1. However, these operations may be executed by a device other than the terminal device 10. For example, the server 30 may include components similar to the display control unit 112 and the operation control unit 114. In particular, in a case where the communication speed between the terminal device 10 and the server 30 is faster and the amount of data transmitted from the server 30 to the terminal device 10 is larger than in a typical information processing system, it is preferable for the server 30 to execute these operations. In the present second modification, the server 30 is an example of an information processing device.

[0087] 2-3: Variation 3 In the terminal device 10 according to the above embodiment, a cylinder Y is assumed as an area in which the line of sight V of the user U1 may exist while the user U1 is viewing the first virtual object VO1. However, the shape of the area is not limited to a cylinder.

[0088] FIG. 10 is an explanatory diagram illustrating an example of the operation of the display control unit 112 according to Modification 3. As shown in FIG. 10, the shape of the above-described region may be, for example, a cone N. The cone N has a central axis L2 with a first endpoint C1 at the position (x0, y0, z0) of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1 and a second endpoint C2 at the position (x1, y1, z1) of the first virtual object VO1. The first endpoint C1 is the apex of the cone N. Furthermore, the radius R2 of the base of the cone N is a length that allows the base to include the first virtual object VO1. Note that, for convenience of explanation, in FIG. 10, the position of the pupil EL of the left eyeball or the pupil ER of the right eyeball of the user U1 does not coincide with the position of the first endpoint C1. Similarly, in FIG. 10, the position (x1, y1, z1) of the first virtual object VO1 does not coincide with the position of the second endpoint C2. However, for practical purposes of the present invention, each position in these sets will coincide.

[0089] 2-4: Variation 4 In the information processing system 1 according to the above embodiment, the terminal device 10 and the MR glasses 20 are realized as separate entities. However, the method for realizing the terminal device 10 and the MR glasses 20 in the embodiment of the present invention is not limited to this. For example, the terminal device 10 and the MR glasses 20 may be realized in a single housing by the MR glasses 20 having the same functions as the terminal device 10.

[0090] 2-5: Variation 5 The information processing system 1 according to the above embodiment includes MR glasses 20. However, the information processing system 1 may include any one of an HMD (Head Mounted Display) employing VR (Virtual Reality) technology, an HMD employing AR (Augmented Reality) technology, and AR glasses employing AR technology, instead of the MR glasses 20. Alternatively, the information processing system 1 may include any one of a regular smartphone and a tablet equipped with an imaging device, instead of the MR glasses 20. These HMDs, AR glasses, smartphones, and tablets are examples of display devices.

[0091] 2-6: Variation 6 In the information processing system 1 according to the above embodiment, the generation unit 312 included in the server 30 generates the first image information indicating the image to be displayed on the MR glasses 20. However, the device that generates the first image information is not limited to the server 30. For example, the terminal device 10 may generate the above first image information. In this case, the information processing system 1 does not need to include the server 30 as an essential component.

[0092] 3:Other (1) In the above-described embodiment, storage device 12, storage device 22, and storage device 32 are exemplified by ROM and RAM, but may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0093] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0094] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0095] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0096] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0097] (6) Each function illustrated in Figures 1 to 10 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0098] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0099] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0100] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0101] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0102] (10) In the above-described embodiments, the terminal device 10 and the server 30 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other appropriate term. In this disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0103] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0104] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0105] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" can include regarding some action as having been "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0106] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0107] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0108] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0109] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0110] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0111] 1...information processing system, 10...terminal device, 11...processing device, 12...storage device, 13...communication device, 14...display, 15...input device, 16...inertial sensor, 20...MR glasses, 21...processing device, 22...storage device, 23...gaze detection device, 24...GPS device, 25...motion detection device, 26...imaging device, 27...communication device, 28...display, 30...server, 31...processing device, 32...storage device, 33...communication device, 34...display, 35...input device, 41L, 41R ...lens, 91, 92...temples, 93...bridge, 94, 95...frame, 111...acquisition unit, 112...display control unit, 113...determination unit, 114...operation control unit, 211...acquisition unit, 212...display control unit, 311...acquisition unit, 312...generation unit, C1, C2...end point, L1, L2...center axis, PR1, PR2, PR3...control program, R1, R2...radius, U1, U2, U3...user, VO...virtual object, VO1...first virtual object, VO2...second virtual object

Claims

1. an acquisition unit that acquires line-of-sight information indicating a line of sight of a user; a display control unit that displays the first virtual object and the second virtual object at positions in the virtual space where they do not overlap each other, as seen by the user; an operation control unit that, after a first state in which the line of sight of the user indicated by the line of sight information intersects with the first virtual object and the first virtual object is active transitions to a second state in which the line of sight of the user does not intersect with the first virtual object, transitions the second virtual object from inactive to active even if the line of sight of the user indicated by the line of sight information does not intersect with the second virtual object; An information processing device comprising:

2. the first virtual object is operated by a first application; the second virtual object is operated by a second application; The first virtual object being active means that the first virtual object is being operated by the first application; transitioning the second virtual object from inactive to active means switching from a state in which the second virtual object is not in operation to a state in which the second virtual object is in operation by the second application; the operation control unit transitions the first virtual object from active to inactive after the first state transitions to the second state; The information processing device according to claim 1 .

3. The information processing device according to claim 1 , wherein the display control unit starts displaying the second virtual object in the virtual space in the first state.

4. 3. The display control unit according to claim 1, wherein, in the second state, the display control unit starts displaying the second virtual object at a position intersecting with the user's line of sight, on condition that a change per unit time of the user's line of sight falls within a first range for a first period of time. Information processing device.

5. 3. The information processing device according to claim 1, wherein the operation control unit transitions the second virtual object from inactive to active on the condition that a change per unit time of the user's line of sight falls within a second range for a second period of time in the second state.

6. The information processing device according to claim 1 , wherein the display control unit causes the second virtual object to be displayed in an area within a first distance from the first virtual object.

7. 3 . The information processing device according to claim 1 , wherein the display control unit displays the second virtual object outside a cylinder or cone in the virtual space, the cylinder or cone having a central axis that coincides with the user's line of sight and a bottom surface that includes the first virtual object.

Images