Display control apparatus

Abstract

To accurately grasp a field of view of a person with whom content is to be shared.SOLUTION: A field-of-view image generation unit 312 generates a field-of-view image FI0 representing a field of view LS2 of a second user U2 in a plane PL1 orthogonal to a line of sight LS1 of a first user U1, based on first field-of-view information related to a field of view FVP1 of the first user U1 and second field-of-view information related to a field of view FVP2 of the second user U2. A display image generation unit 313 generates, based on the first field-of-view information, a display image DI0 formed by arranging the field-of-view image FI0 in the field of view FVP1 of the first user U1. A display control unit 314 displays the display image DI0 on an MR glass 20a of the first user U1.SELECTED DRAWING: Figure 18

Description

[Technical field]

[0001] The present invention relates to a display control device. [Background technology]

[0002] Conventionally, in a situation where two or more users wearing XR glasses share content in a virtual space, it is difficult for the two or more users to grasp each other's field of view. Patent Document 1 discloses an information processing device in which, when multiple users view the same content, a first field of view image corresponding to the field of view of a first user is displayed on a content image, and a second field of view image corresponding to the field of view of a second user is displayed on the content image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-13016 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional information processing devices have a problem in that it is difficult for two or more users to accurately inform each other of the location of content that they wish to share.

[0005] The present invention has been made to solve the above-mentioned problems, and has an object to accurately grasp the other party's field of view in order to accurately inform the other party of the location of content to be shared. [Means for solving the problem]

[0006] A display control device according to a preferred embodiment of the present invention includes a field of view image generation unit that generates a field of view image showing a field of view of a second user in a plane perpendicular to the line of view of a first user based on first field of view information regarding the field of view of the first user including the line of view of the first user and second field of view information regarding the field of view of a second user, a display image generation unit that generates a display image in which the field of view image is arranged in the field of view of the first user based on the first field of view information, and a display control unit that displays the display image on a glass device worn on the head of the first user. Effect of the Invention

[0007] According to the display control device of the present invention, it is possible to accurately grasp the other party's field of vision. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overall configuration of an information processing system including a display control device according to a first embodiment. [Diagram 2] FIG. 2 is a perspective view showing the appearance of MR glasses. [Diagram 3] 1 is a schematic diagram of a virtual space displayed by MR glasses. [Figure 4] 2 is a block diagram showing an example of the configuration of the MR glasses of FIG. 1. [Diagram 5] 2 is a block diagram showing an example of the configuration of a terminal device in FIG. 1. [Figure 6] 2 is a block diagram showing an example of the configuration of a server in FIG. 1. [Figure 7] FIG. 11 is a schematic diagram for explaining the field of view of a first user. [Figure 8] FIG. 11 is a schematic diagram for explaining the field of view of a second user. [Figure 9] 1 is a schematic diagram for explaining the relationship between a field of view of a first user and a field of view of a second user. FIG. [Figure 10] FIG. 11 is a schematic diagram for explaining a field of view image of a second user. [Figure 11] FIG. 13 is a diagram showing an example of a display image. [Figure 12]11 is a diagram showing the relationship between the field of view of a first user and the field of view of a second user when the point of gaze is located outside the field of view of the second user. FIG. [Figure 13] 13 is a diagram showing an example of a display image displayed on the MR glasses of a first user in the case shown in FIG. 12. FIG. [Figure 14] 11 is a diagram showing the relationship between the field of view of a first user and the field of view of a second user when the point of gaze is located at the edge of the field of view of the second user. FIG. [Figure 15] 15 is a diagram showing an example of a display image displayed on the MR glasses of a first user in the case shown in FIG. 14. FIG. [Figure 16] 11 is a diagram showing the relationship between the field of view of a first user and the field of view of a second user when the point of gaze is located within the field of view of the second user. FIG. [Figure 17] 17 is a diagram showing an example of a display image displayed on the MR glasses of a first user in the case shown in FIG. 16. FIG. [Figure 18] 7 is a flowchart showing the operation of the processing device in FIG. 6. [Figure 19] FIG. 11 is a diagram showing an overall configuration of an information processing system including a display control device according to a second embodiment. [Figure 20] 20 is a block diagram showing an example of the configuration of a server in FIG. 19. [Figure 21] FIG. 11 is a schematic diagram for explaining the field of view of a first user. [Figure 22] 21 is a flowchart showing the operation of the processing device in FIG. 20. [Diagram 23] 13 is a schematic diagram for explaining a visual field image of a second user according to Modification 1. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1. First embodiment Hereinafter, the configuration of a display control device according to a first embodiment of the present invention will be described with reference to FIGS.

[0010] 1.1. Configuration of the First Embodiment Overall Configuration 1 is a diagram showing an overall configuration of an information processing system 1 including a display control device according to a first embodiment. The information processing system 1 includes a terminal device 10, MR glasses 20, a server 30, and a communication network NET. The MR glasses 20 are an example of a glass device, and the server 30 is an example of a display control device.

[0011] The terminal device 10 includes two terminal devices 10a and 10b. The MR glasses 20 include two MR glasses 20a and 20b. A first user U1 corresponds to the terminal device 10a and the MR glasses 20a. A second user U2 corresponds to the terminal device 10b and the MR glasses 20b. The first user U1 carries the terminal device 10a and wears the MR glasses 20a on his / her head. The second user U2 carries the terminal device 10b and wears the MR glasses 20b on his / her head.

[0012] In this embodiment, the configurations of the terminal devices 10a and 10b are the same as each other, and the configurations of the MR glasses 20a and 20b are the same as each other. The terminal device 10 may include terminal devices that are not the same in configuration, and the MR glasses 20 may include MR glasses that are not the same in configuration. In addition, in FIG. 1, two pairs of the terminal device 10 and the MR glasses 20 are illustrated as an example, but the number of pairs is merely an example, and the information processing system 1 can include any number of pairs of the terminal device 10 and the MR glasses 20.

[0013] The terminal device 10 causes the MR glasses 20 to display virtual objects arranged in a virtual space. The MR glasses 20 are wearable displays worn on the user's head. The MR glasses 20 display virtual objects on display panels provided on the lenses corresponding to the user's eyes, respectively, according to the control of the terminal device 10. Each lens and display panel of the MR glasses 20 are see-through. Therefore, a user wearing the MR glasses 20 can view the real space through each lens and display of the MR glasses 20.

[0014] The terminal device 10 includes a personal computer, a tablet terminal, a smartphone, a smart watch, and the like.

[0015] In the information processing system 1, terminal devices 10a and 10b and a server 30 are communicably connected to each other via a communication network NET.

[0016] The server 30 is a server that provides an image display service. The server 30 collects various information related to the image display service from the terminal device 10 via the communication network NET, and manages events based on the various information.

[0017] The information processing system 1 is a system that provides an image display service to any user who has a terminal device 10 and wears MR glasses 20.

[0018] 1.1.2. Composition of MR glasses Fig. 2 is a perspective view showing the appearance of the MR glasses 20. As shown in Fig. 2, the MR glasses 20 have a left temple 91, a right temple 92, a bridge 93, a left frame 94, a right frame 95, and a left eye lens 41L and a right eye lens 41R, similar to general eyeglasses. An imaging device 26 is provided on the bridge 93. The imaging device 26 captures an image of the outside world. The imaging device 26 also outputs imaging information indicating the captured image.

[0019] Although not shown, the MR glasses 20 are provided with a motion detection device that detects the motion of the user's head. The MR glasses 20 are provided with a geomagnetic sensor, an acceleration sensor, and a gyro sensor as sensors that detect the motion of the user. The MR glasses 20 are provided with a left eye display and a right eye display as output devices, i.e., displays.

[0020] The left eye display has a left eye lens 41L, a left eye display panel, and a left eye optical member. As the left eye display panel, various display panels such as a liquid crystal panel and an organic EL panel are suitably used. The left eye optical member includes a light guide and a half mirror. The light guide guides light emitted from the left eye display panel to the half mirror. The half mirror transmits light from the outside world to guide the light from the outside world to the left eye of the user, while reflecting light from the light guide to guide the light emitted from the left eye display panel to the left eye of the user.

[0021] The right-eye display has a right-eye lens 41R, a right-eye display panel, and a right-eye optical member. The right-eye display panel uses a display panel similar to that of the left-eye display panel. The right-eye optical member includes a light guide and a half mirror similar to those of the left-eye optical member. The light guide guides light emitted from the right-eye display panel to the half mirror. The half mirror transmits light from the outside world to guide the light from the outside world to the right eye of the user, while reflecting light from the light guide to guide the light emitted from the right-eye display panel to the right eye of the user.

[0022] With the above configuration, the user can wear the MR glasses 20 to observe the images displayed on the left eye display panel and the right eye display panel superimposed on the outside world. In addition, the MR glasses 20 display the left eye image on the left eye display panel and the right eye image on the right eye display panel, among the binocular images with parallax. This allows the MR glasses 20 to make the user perceive the displayed image as if it has depth and three-dimensionality.

[0023] Fig. 3 is a schematic diagram of a virtual space displayed by the MR glasses 20. As shown in Fig. 3, virtual objects VO1 to VO5 are arranged in this virtual space. In this embodiment, each of the virtual objects VO1 to VO5 is arranged on a spherical surface R of a predetermined radius centered on the position of the user's head. Here, since the spherical surface R is transparent, the user can see through the spherical surface R and thus view the outside world outside the spherical surface R.

[0024] The virtual objects are, for example, virtual objects representing data such as a still image, a video, a three-dimensional CG model, an HTML file, a text file, and a virtual object representing an application. Here, examples of text files include memos and source code. Examples of applications include a browser, an application for using an SNS, and an application for generating a document file. Note that the number of virtual objects VO in FIG. 3 is merely an example, and the number of virtual objects VO is not limited to the present disclosure.

[0025] The terminal device 10 includes a personal computer, a tablet terminal, a smartphone, and a smart watch. The terminal device 10 is preferably a mobile terminal device such as a tablet terminal or a smartphone. The MR glasses 20 may be connected directly to the communication network NET without going through the terminal device 10.

[0026] The user can experience the virtual space as a private space in a public space by moving around the public space while wearing the MR glasses 20 on which the virtual objects VO1 to VO5 in the virtual space are displayed. The user can act in the public space while enjoying the benefits brought about by the virtual objects VO1 to VO5 arranged in the virtual space.

[0027] Each user holds a terminal device 10. Each user uses the terminal device 10 to perform operations on the MR glasses 20 or the server 30.

[0028] Fig. 4 is a block diagram showing an example of the configuration of the MR glasses 20 in Fig. 1. The MR glasses 20 include a processing device 21, a storage device 22, a line-of-sight acquisition device 23, a positioning device 24, a motion detection device 25, an imaging device 26, a communication device 27, and a display 28. The elements of the MR glasses 20 are connected to each other by a single or multiple buses for communicating information. Note that the term "apparatus" in this specification may be replaced with other terms such as a circuit, a device, or a unit.

[0029] The processing device 21 is a processor that controls the entire MR glasses 20, and is configured, for example, using a single chip or multiple chips. The processing device 21 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 21 may be realized by hardware such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). The processing device 21 executes various processes in parallel or sequentially.

[0030] The storage device 22 is a recording medium that can be read and written by the processing device 21. The storage device 12 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM).

[0031] The storage device 22 stores a plurality of programs including a control program PR2 to be executed by the processing device 21. The storage device 22 also functions as a work area for the processing device 21.

[0032] The gaze acquisition device 23 acquires the direction in which the user is looking, i.e., the direction of the user's gaze, by eye tracking, which tracks the movement of the user's left and right eyes, and outputs gaze information indicating the direction of the user's gaze to the processing device 21 based on the acquired results.

[0033] More specifically, the line-of-sight acquisition device 23 includes a pair of light sources and a pair of cameras. Each of the pair of light sources corresponds to the left eye and the right eye of the user, and each of the pair of cameras corresponds to the left eye and the right eye of the user. The light source corresponding to the left eye of the user irradiates infrared light to the left eye of the user, and the camera corresponding to the left eye of the user captures a corneal reflection image of the left eye of the user as an image formed by reflection of the irradiated infrared light. Similarly, the light source corresponding to the right eye of the user irradiates infrared light to the right eye of the user, and the camera corresponding to the right eye of the user captures a corneal reflection image of the right eye of the user.

[0034] The gaze acquisition device 23 detects the position of the inner corner of the user's eye and the position of the iris from the acquired corneal reflection image of the left eye and the corneal reflection image of the right eye, and acquires gaze information of the user based on the detected positions of the inner corner of the user's eye and the iris. The gaze information output to the processing device 21 is output to the terminal device 10 via the communication device 27. Note that the method of acquiring the gaze by the gaze acquisition device 23 is not limited to the above method, and any method may be used.

[0035] The positioning device 24 acquires position information of the MR glasses 20. The positioning device 24 may be, for example, a GNSS (Global Navigation Satellite System) receiver. The GNSS receiver receives radio signals transmitted from one or more GNSS satellites. The GNSS is a positioning system using positioning satellites from countries around the world, including GPS (Global Positioning System) satellites. The radio signals include information such as position information of the satellite that transmitted the radio signals and the transmission time of the radio signals. The GNSS receiver performs positioning based on the one or more received radio signals, and outputs the position information of the GNSS receiver to the processing device 21. The position information indicates, for example, the latitude and longitude of the MR glasses 20.

[0036] The positioning device 24 may be, for example, a VPS (Visual Positioning Service) device. The VPS device acquires image information indicating an image of a scene in front of the user from the imaging device 26. The VPS device outputs the image information acquired from the imaging device 26 to a position information server (not shown) via the communication device 27. The VPS device acquires VPS information as position information from the position information server via the communication device 27. The position information includes the position of the user in real space and the direction in which the user views the real space.

[0037] The MR glasses 20 may acquire the position information by any method. The position information may be in any format as long as the position can be specified. The acquired position information is output to the processing device 21. The position information output to the processing device 21 is output to the terminal device 10 via the communication device 27.

[0038] The motion detection device 25 detects the motion of the MR glasses 20 and outputs motion data to the processing device 21. The motion data includes acceleration data indicating the acceleration in each of the X-axis, Y-axis, and Z-axis directions, and angular acceleration data indicating the angular acceleration about each of the X-axis, Y-axis, and Z-axis as the center of rotation. The motion detection device 25 includes an acceleration sensor that detects acceleration, an inertial sensor such as a gyro sensor that detects angular acceleration, and a geomagnetic sensor that detects the direction in which the MR glasses 20 are facing.

[0039] The acceleration sensor detects the acceleration in each of the mutually orthogonal X-axis, Y-axis, and Z-axis directions. The gyro sensor detects the angular acceleration around the X-axis, Y-axis, and Z-axis as the center of rotation. The geomagnetic sensor detects the geomagnetism in each of the X-axis, Y-axis, and Z-axis directions, and detects the direction in which the MR glasses 20 are facing. The motion detection device 25 generates attitude information indicating the attitude of the MR glasses 20 based on the output information from the gyro sensor. The motion detection device 25 also outputs motion information related to the motion of the MR glasses 20 to the processing device 21. The motion information output to the processing device 21 is output to the terminal device 10 via the communication device 27.

[0040] The imaging device 26 outputs imaging information obtained by imaging the outside world. The imaging device 26 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 21. The imaging information output to the processing device 21 is output to the terminal device 10 via the communication device 27.

[0041] As a result, in the terminal device 10, various gestures of the user are recognized based on the imaging information by the imaging device 26, and control is executed according to the recognized gestures. That is, the imaging device 26 functions as an input device for inputting user instructions, like a pointing device and a touch panel.

[0042] The communication device 27 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 27 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 27 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 27 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, USB, etc. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0043] The display 28 is a device that displays images. The display 28 displays various images under the control of the processing device 21. The display 28 includes a display for the left eye and a display for the right eye.

[0044] The processing device 21 functions as an acquisition unit 211 and a display control unit 212, for example, by reading out a control program PR2 from the storage device 22 and executing it.

[0045] The acquisition unit 211 acquires a control signal and image information from the terminal device 10. The control signal is a signal for controlling display on the display 28. The control signal is generated by a display control unit 113 (described later) included in the terminal device 10. The image information is information indicating an image to be displayed on the display 28. The image information is information including information indicating virtual objects VO1 to VO5. The image information may be stored in the storage device 12 of the terminal device 10, or may be stored in a storage device 32 (described later) of the server 30.

[0046] Furthermore, the acquisition unit 211 acquires gaze information input from the gaze acquisition device 23, position information input from the positioning device 24, movement information input from the movement detection device 25, and imaging information input from the imaging device 26. Then, the acquisition unit 211 outputs the acquired gaze information, position information, movement information, and imaging information to the communication device 27.

[0047] The display control unit 212 controls the display on the display 28, based on a control signal from the terminal device 10 acquired by the acquisition unit 211. In addition, the display control unit 212 causes the display 28 to display the virtual object VO, based on the image information acquired by the acquisition unit 211.

[0048] 1.1.3. Terminal Device Configuration Fig. 5 is a block diagram showing an example of the configuration of the terminal device 10 in Fig. 1. As shown in Fig. 5, the terminal device 10 includes a processing device 11, a storage device 12, a communication device 13, a display 14, an input device 15, and an imaging device 16. The elements included in the terminal device 10 are connected to each other by a single or multiple buses for communicating information.

[0049] The processing device 11 is a processor that controls the entire terminal device 10, and is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, an ASIC, a PLD, an FPGA, etc. The processing device 11 executes various processes in parallel or sequentially.

[0050] The storage device 12 is a recording medium that can be read and written by the processing device 11. The storage device 12 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0051] The storage device 12 stores a plurality of programs including a control program PR1 to be executed by the processing device 11. The storage device 12 also functions as a work area for the processing device 11.

[0052] The communication device 13 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0053] The display 14 is a device that displays images and text information. The display 14 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal panel and an organic EL panel are suitably used as the display 14.

[0054] The input device 15 receives operations from a user. For example, the input device 15 includes a keyboard, a touch pad, a touch panel, and a pointing device such as a mouse. Here, when the input device 15 includes a touch panel, it may also serve as the display 14.

[0055] The imaging device 16 outputs an imaged image Gx obtained by capturing an image of the outside world. The imaging device 16 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 11 as an imaged image Gx.

[0056] The processing device 11 functions as an acquisition unit 111, an output unit 112, and a display control unit 113, for example, by reading out a control program PR1 from the storage device 12 and executing it.

[0057] The acquisition unit 111 acquires instruction information according to an operation of a user wearing the MR glasses 20 on the head. Here, the user's operation is, for example, an operation input by the user to the terminal device 10 using the input device 15. More specifically, the user's operation may be pressing a specific part of the input device 15. Alternatively, the user's operation may be an operation using the terminal device 10 as a pointing device or a portable controller.

[0058] The acquisition unit 111 acquires image information indicating the virtual objects VO1 to VO5 from the server 30. The acquisition unit 111 also acquires instruction information (operation information) from the input device 15. The acquisition unit 111 also acquires position information from the positioning device 24 of the MR glasses 20.

[0059] The acquisition unit 111 acquires image information from the server 30 by using the communication device 13. The acquisition unit 111 stores the acquired image information in the storage device 12. The acquisition unit 111 acquires the image information stored in the storage device 12.

[0060] The output unit 112 outputs the position information and the user's visibility information to the server 30 via the communication device 13 as necessary.

[0061] The display control unit 113 causes the display 14 to display various pieces of information based on the various pieces of information acquired by the acquisition unit 111 .

[0062] 1.1.4. Server configuration Fig. 6 is a block diagram showing an example of the configuration of the server 30 in Fig. 1. As shown in Fig. 6, the server 30 includes a processing device 31, a storage device 32, a communication device 33, a display 34, and an input device 35. The elements included in the server 30 are connected to each other by one or more buses for communicating information.

[0063] The processing device 31 is a processor that controls the entire server 30, and is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 31 may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 31 executes various processes in parallel or sequentially.

[0064] The storage device 32 is a recording medium that can be read and written by the processing device 31. The storage device 32 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0065] The storage device 32 stores a plurality of programs including a control program PR3 to be executed by the processing device 31. The storage device 32 also functions as a work area for the processing device 31.

[0066] The communication device 33 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0067] The display 34 is a device that displays images and text information. The display 34 displays various images under the control of the processing device 31. For example, various display panels such as a liquid crystal panel and an organic EL panel are suitably used as the display 34.

[0068] The input device 35 is a device that accepts operations by an administrator of the information processing system 1. For example, the input device 35 includes a keyboard, a touch pad, a touch panel, and a pointing device such as a mouse. Here, when the input device 35 includes a touch panel, it may also function as the display 34. The administrator of the information processing system 1 can modify the control program PR3 by using the input device 35.

[0069] The processing device 31 functions as an acquisition unit 311, a field of view image generation unit 312, a display image generation unit 313, and a display control unit 314, for example, by reading and executing a control program PR3 from the storage device 32.

[0070] 7 is a schematic diagram for explaining the field of view of the first user U1. As shown in FIG. 7, the first user U1 wearing the MR glasses 20a on his / her head can view the real space through the MR glasses 20a. The field of view range FV1 of the first user U1 is a quadrangular pyramid-shaped range surrounded by four boundary lines extending radially from the apex of the head, that is, the upper left end of the field of view FV1UL, the upper right end of the field of view FV1UR, the lower right end of the field of view FV1LR, and the lower left end of the field of view FV1LL. The three-dimensional space coordinates of the MR glasses 20a are (xu1, yu1, zu1).

[0071] The gaze point, which is the point at which the first user U1 is gazing, is defined as P1, and the three-dimensional space coordinates of the gaze point P1 are defined as (xp1, yp1, zp1). A line extending from the MR glasses 20a to the gaze point P1 is the line of sight LS1 of the first user U1. At the gaze point P1, a plane PL1 perpendicular to the line of sight LS1 of the first user U1 is cut into a rectangle in a quadrangular pyramid-shaped range surrounded by the upper left end FV1UL of the field of view, the upper right end FV1UR of the field of view, the lower right end FV1LR of the field of view, and the lower left end FV1LL of the field of view. In this embodiment, the plane PL1 cut into a rectangle is defined as the field of view FVP1 of the first user U1 based on the gaze point P1.

[0072] The shape of the field of view FVP1 of the first user U1 based on the gaze point P1 does not have to be rectangular. Depending on the structure, shape, etc. of the MR glasses 20, the shape of the field of view FVP1 of the first user U1 can be any shape.

[0073] Fig. 8 is a schematic diagram for explaining the field of view of the second user U2. As shown in Fig. 8, the second user U2 wearing the MR glasses 20b on his / her head can view the real space through the MR glasses 20b. The field of view range FV2 of the second user U2 is a quadrangular pyramid-shaped range that has the MR glasses 20b as the apex and is surrounded by four boundary lines that extend radially from the apex, that is, the upper left end FV2UL of the field of view, the upper right end FV2UR of the field of view, the lower right end FV2LR of the field of view, and the lower left end FV2LL of the field of view.

[0074] The second user U2 is located far enough away from the first user U1 that the second user U2 can share a landmark in real space with the first user U1, and the three-dimensional space coordinates of the MR glasses 20b are (xu2, yu2, zu2). Here, sharing a landmark means that the first user U1 and the second user U2 can visually recognize a common landmark. The first user U1 and the second user U2 may be located close to each other. Here, each user and the landmark are far enough away that the distance between each user and the landmark is sufficiently longer than the distance between the first user U1 and the second user U2.

[0075] The gaze point, which is the point that the second user U2 is gazing at, is defined as P2, and the three-dimensional space coordinates of the gaze point P2 are defined as (xp2, yp2, zp2). A line extending from the MR glasses 20b toward the gaze point P2 is the line of sight LS2 of the second user U2. At the gaze point P2, a plane PL2 perpendicular to the line of sight LS2 of the second user U2 is cut into a rectangle in a quadrangular pyramid-shaped range surrounded by the upper left end FV2UL of the field of view, the upper right end FV2UR of the field of view, the lower right end FV2LR of the field of view, and the lower left end FV2LL of the field of view. In this embodiment, the plane PL2 cut into a rectangle is defined as the field of view FVP2 of the second user U2 based on the gaze point P2. Note that the distance between the first user U1 and the gaze point P1 and the distance between the second user U2 and the gaze point P2 are set to a sufficiently long predetermined distance.

[0076] Fig. 9 is a schematic diagram for explaining the relationship between the field of view of the first user U1 and the field of view of the second user U2. As shown in Fig. 9, the first user U1 and the second user U2 wear MR glasses 20a and 20b, respectively, and gaze at different gaze points P1 and P2. In addition, a part of the field of view FVP1 of the first user U1 and a part of the field of view FVP2 of the second user U2 overlap with each other at an overlapping part OL1.

[0077] For example, consider that a first user U1 shares a gaze point P1 with a second user U2. More specifically, the first user U1 wants the second user U2 to pay attention to the gaze point P1. To do so, the first user U1 needs to grasp the gaze point P2 of the second user U2 or the field of view FVP2 of the second user U2, and to grasp the positional relationship between the gaze point P1 of the first user U1 and the gaze point P2 of the second user U2 or the field of view FVP2 of the second user U2.

[0078] If the first user U1 can visually recognize the overlap OL1 between a part of the field of view FVP1 of the first user U1 and a part of the field of view FVP2 of the second user U2, the first user U1 can grasp the positional relationship between the gaze point P1 of the first user U1 and the field of view FVP2 of the second user U2. Therefore, in this embodiment, as described in detail below, the first user U1 can display the overlap OL1 on the display 28 of the MR glasses 20a as a field of view image of the second user U2 by performing a predetermined operation of the terminal device 10a.

[0079] The same applies to the case where the second user U2 shares the gaze point P2 with the first user U1. That is, the second user U2 can display the overlap OL1 as the field of view image of the first user U1 on the display 28 of the MR glasses 20b by performing a predetermined operation on the terminal device 10b. In this embodiment, a case where the field of view image of the second user U2 is displayed on the MR glasses 20a of the first user U1 will be described.

[0080] Fig. 10 is a schematic diagram for explaining a field of view image FI0 of a second user U2. Fig. 10 shows a first user U1, a field of view FVP1 of the first user U1, and a part of a field of view FVP2 of the second user U2.

[0081] Here, when the first user U1 performs a predetermined operation using the terminal device 10a, the output unit 112 transmits operation information regarding the predetermined operation by the first user U1 to the server 30. For example, the predetermined operation includes an operation to indicate a desire to share the view of another user and an operation to select an account or nickname of a user with whom the view can be shared. Here, the first user U1 selects the account of the second user U2.

[0082] The communication device 33 of the server 30 receives the operation information. The acquisition unit 311 acquires the operation information received by the communication device 33, and acquires the first visibility information from the terminal device 10a and the second visibility information from the terminal device 10b based on the operation information.

[0083] The first field of view information is information about the field of view FVP1 of the first user U1. The first field of view information includes information about the position (xu1, yu1, zu1) of the MR glasses 20a and information about the position (xp1, yp1, zp1) of the gaze point P1 of the first user U1. The first field of view information also includes information about the line of sight LS1 of the first user U1 and information about the field of view range FV1 of the first user U1.

[0084] The second field of view information is information about the field of view FVP2 of the second user U2. The second field of view information includes information about the position (xu2, yu2, zu2) of the MR glasses 20b and information about the position (xp2, yp2, zp2) of the gaze point P2 of the second user U2. The second field of view information also includes information about the line of sight LS2 of the second user U2 and information about the field of view range FV2 of the second user U2.

[0085] The field of view image generating unit 312 generates a field of view image FI0 based on the first field of view information and the second field of view information. The field of view image FI0 is an image that includes a gaze point P1 on the line of sight LS1 of the first user U1 and shows the field of view of the second user U2 on a plane PL1 perpendicular to the line of sight LS1 of the first user U1. A method of generating the field of view image FI0 based on the first field of view information and the second field of view information will be described in detail later.

[0086] The display image generating unit 313 generates a display image DI0 based on the first field of view information. FIG. 11 is a diagram showing an example of the display image DI0. As shown in FIG. 11, the display image DI0 is an image in which the field of view image FI0 generated by the field of view image generating unit 312 is arranged in the field of view of the first user U1. The display image DI0 is hatched as a whole so that it is easy for the first user U1 to view. Although the display image DI0 is hatched, it is preferable that it is transparent so that the first user U1 can view the real space through the display image DI0.

[0087] The display control unit 314 displays the display image DI0 generated by the display image generating unit 313 on the MR glasses 20a.

[0088] 1.2. Generation of display image by the display control device according to the first embodiment Hereinafter, a procedure for generating a display image DI0 to be displayed on the MR glasses 20a worn on the head of the first user U1 will be described with reference to FIGS.

[0089] 1.2.1. When the gaze point P1 is located outside the field of view of the second user U2 Fig. 12 is a diagram showing the relationship between the field of view FVP1 of the first user U1 and the field of view FVP2 of the second user U2 when the gaze point P1 is located outside the field of view of the second user U2. Fig. 13 is a diagram showing an example of a display image DI0 displayed on the MR glasses 20a of the first user U1 in the case shown in Fig. 12. Fig. 12 is a schematic diagram of the first user U1, the line of sight LS1 of the first user U1, the field of view range FV1 of the first user U1, the gaze point P1, the second user U2, the line of sight LS2 of the second user U2, and the field of view range FV2 of the second user U2 viewed from above.

[0090] As shown in Fig. 12, the x-axis is provided along a straight line connecting the first user U1 and the second user U2. For convenience, the x-axis is an axis on a horizontal plane. The distance between the first user U1 and the second user U2 is set to d. The distance between the first user U1 and the fixation point P1 on the horizontal plane is set to ru1, and the distance between the second user U2 and the fixation point P1 on the horizontal plane is set to ru2.

[0091] In this example, the first user U1 is gazing at the gaze point P1. Therefore, the line of sight LS1 of the first user U1 is directed toward the gaze point P1. The field of view FV1 of the first user U1 ranges from the left end FV1L of the field of view of the first user U1 to the right end FV1R of the field of view of the first user U1. The field of view FV2 of the second user U2 ranges from the left end FV2L of the field of view of the second user U2 to the right end FV2R of the field of view of the second user U2. The line of sight LS2 of the second user U2 is in the middle of the field of view range of the second user U2. Therefore, in this example, the field of view of the second user U2 does not include the gaze point P1 of the first user U1.

[0092] The circle C1 is a circle with a radius ru1 centered on the first user U1. The circle C2 is a circle with a radius ru2 centered on the second user U2. The circles C1 and C2 intersect at the position of the gaze point P1.

[0093] The angle between the x-axis and the line of sight LS1 of the first user U1 is θu1, and the angle between the x-axis and the line of sight LS2 of the second user U2 is θu2. The angle between the right edge FV2R of the field of view of the second user U2 as seen from the first user U1 and the direction of the gaze point P1 is φu1, and the angle between the right edge FV2R of the field of view of the second user U2 and the direction of the gaze point P1 as seen from the second user U2 is φu2.

[0094] The distance Lu1 between the intersection point K1 between the right edge FV2R of the field of view of the second user U2 as seen by the first user U1 and the circle C1 and the point of gaze P1 is approximated by the length of the arc from the intersection point K1 to the point of gaze P1, and is expressed by the following equation (1). Lu1=2π×ru1×φu1 / 360 ···(1)

[0095] The distance Lu2 between the intersection point K2 between the right edge FV2R of the field of view of the second user U2 and the circle C2 and the gaze point P1 as seen by the second user U2 is approximated by the length of the arc from the intersection point K2 to the gaze point P1, and is expressed by the following equation (2). Lu2=2π×ru2×φu2 / 360 ···(2)

[0096] If the field of view of the first user U1, i.e., the angle between the left edge FV1L of the field of view of the first user U1 and the right edge FV1R of the field of view of the first user U1, is α, the horizontal length LFV1 of a plane perpendicular to the line of sight LS1 of the first user U1, including the gaze point P1, is approximated by the following equation (3): LFV1=2π×ru1×α / 360 (3)

[0097] The ratio λ(L) of Lu1 to the length LFV1 is expressed by the following formula (4). λ(L)=Lu1 / LFV1 (4)

[0098] Therefore, the range of the display image DI0 displayed in the field of view FVP1 of the first user U1 is the range from the left end of the field of view FVP1 of the first user U1 to a position moved to the left of the field of view FVP1 of the first user U1 by a ratio λ(L) from the gaze point P1 to the left end of the field of view FVP1 of the first user U1, as shown in FIG. 13.

[0099] In this way, since the display image DI0 is displayed in a portion of the left side of the field of view FVP1 of the first user U1, the first user U1 can understand that the second user U2 is directing his / her gaze LS2 to the left of the gaze point P1.

[0100] 1.2.2. When the gaze point P1 is located at the edge of the field of view of the second user U2 Fig. 14 is a diagram showing the relationship between the field of view FVP1 of the first user U1 and the field of view FVP2 of the second user U2 when the gaze point P1 is located at the edge of the field of view of the second user U2. Fig. 15 is a diagram showing an example of a display image DI0 displayed on the MR glasses 20a of the first user U1 in the case shown in Fig. 14. As shown in Fig. 14, the right edge FV2R of the field of view of the second user U2 faces the gaze point P1 of the second user U2. Similarly to Fig. 12, the line of sight LS1 of the first user U1 also faces the gaze point P1.

[0101] 14, the angle φu1 between the right edge FV2R of the field of view of the second user U2 and the direction of the gaze point P1 as seen by the first user U1 is 0. Similarly, the angle φu2 between the right edge FV2R of the field of view of the second user U2 and the direction of the gaze point P1 as seen by the second user U2 is 0. Therefore, from equation (1), Lu1 is 0, and from equation (2), Lu2 is 0.

[0102] From the above, when the gaze point P1 is located at the edge of the field of view of the second user U2, λ(L) is 0 according to equation (4).

[0103] Therefore, the range of the display image DI0 displayed in the field of view of the first user U1 is the range from the left end of the field of view FVP1 of the first user U1 to the gaze point P1, as shown in FIG.

[0104] In this way, since the display image DI0 is displayed in the left half of the field of view FVP1 of the first user U1, the first user U1 can recognize that the right edge of the field of view of the second user U2 is located at the gaze point P1.

[0105] 1.2.3. When the gaze point P1 is located within the field of view of the second user U2 FIG. 16 is a diagram showing the relationship between the field of view FVP1 of the first user U1 and the field of view FVP2 of the second user U2 when the gaze point P1 is located within the field of view of the second user U2. FIG. 17 is a diagram showing an example of a display image DI0 displayed on the MR glasses 20a of the first user U1 in the case shown in FIG. 16. As shown in FIG. 16, the right end FV2R of the field of view of the second user U2 intersects with the right end FV1R of the field of view of the first user U1. The line of sight LS2 of the second user U2 is directed toward the gaze point P1. Also, similar to the case shown in FIG. 12, the line of sight LS1 of the first user U1 is directed toward the gaze point P1.

[0106] In this case, φu1 is the angle between the right edge FV1R of the field of view of the first user U1 and the direction of the gaze point P1. In Fig. 16, the right edge FV1R of the field of view of the first user U1 and the right edge FV2R of the field of view of the second user U2 intersect on the circle C2, and the right edge FV2R of the field of view of the second user U2 as seen from the first user U1 covers the right edge FV1R of the field of view of the first user U1. Therefore, φu1 can be rephrased as the angle between the right edge FV2R of the field of view of the second user U2 as seen from the first user U1 and the direction of the gaze point P1.

[0107] φu2 is the angle between the right edge FV2R of the field of view of the second user U2 and the direction of the gaze point P1 as seen by the second user U2.

[0108] 17, the range of the display image DI0 displayed in the field of view FVP1 of the first user U1 is the range from the left end of the field of view FVP1 of the first user U1 to the point of gaze P, plus the range expanded from the point of gaze P1 to the right of the field of view FVP1 of the first user U1 by a ratio λ(L). In this case, the ratio λ(L) is 0.5. That is, the range of the display image DI0 displayed in the field of view FVP1 of the first user U1 is the entire field of view FVP1 of the first user U1.

[0109] In this way, the display image DI0 is displayed in the entire field of view FVP1 of the first user U1, so that the first user U1 can understand that the second user U2 is directing the line of sight LS2 toward the gaze point P1.

[0110] 1.3. Operation of the display control device according to the first embodiment 1.3.1. First Operation of Processing Device 31 Fig. 18 is a flowchart showing the operation of the processing device 31 in Fig. 6. Hereinafter, the operation of the processing device 31 will be described with reference to Fig. 18. The routine in Fig. 18 is started, for example, when the processing device 31 is started, and is executed every time a certain period of time has elapsed.

[0111] In step S11, the processing device 31 functions as the acquisition unit 311 to determine whether or not operation information has been acquired from the terminal device 10a.

[0112] If no operation information is acquired from the terminal device 10a, that is, if the determination result in step S11 is negative, the processor 31 temporarily ends this routine.

[0113] On the other hand, when operation information is acquired from the terminal device 10a, i.e., when the judgment result in step S11 is positive, the processing device 31 functions as an acquisition unit 311, and in step S12, acquires the first visibility information from the terminal device 10a and the second visibility information from the terminal device 10b.

[0114] In step S13, the processing device 31 functions as the field of view image generating unit 312 to identify a plane PL1 that includes the position of the gaze point P1 of the first user U1 and is perpendicular to the line of sight LS1 of the first user U1.

[0115] In step S14, the processing device 31 functions as the field of view image generating unit 312 to generate a field of view image FI0 of the second user U2 based on the first field of view information and the second field of view information.

[0116] In step S15, the processing device 31 functions as the display image generating section 313 to generate a display image DI0 based on the first visibility information.

[0117] In step S16, the processing device 31 functions as the display control unit 314 to display the display image DI0 generated by the display image generation unit 313 on the MR glasses 20a worn on the head of the first user U1, and then temporarily terminates this routine.

[0118] 1.4. Advantages of the First Embodiment According to the above description, the processing device 31 of the server 30 according to the first embodiment includes a field of view image generating unit 312, a display image generating unit 313, and a display control unit 314. The field of view image generating unit 312 generates a field of view image FI0 indicating the field of view FVP2 of the second user U2 in a plane PL1 perpendicular to the line of sight LS1 of the first user U1 based on the first field of view information and the second field of view information. The first field of view information is information related to the field of view FVP1 of the first user U1. The first field of view information includes information on the line of sight LS1 of the first user U1. The second field of view information is information related to the field of view FVP2 of the second user U2.

[0119] The display image generation unit 313 generates a display image DI0 in which the field of view image FI0 is arranged in the field of view FVP1 of the first user U1 based on the first field of view information. The display control unit 314 displays the display image DI0 on the MR glasses 20a worn on the head of the first user U1.

[0120] According to this embodiment, the field of view image FI0 of the second user U2 can be obtained based on the first field of view information and the second field of view information, so that the positional relationship between the field of view FVP1 of the first user U1 and the field of view FVP2 of the second user U2 can be obtained more accurately. Therefore, the first user U1 can accurately grasp the field of view of the second user U2.

[0121] The first field of view information also includes the position of the gaze point P1 of the first user U1. The field of view image generation unit 312 specifies a plane PL1 that includes the gaze point P1 of the first user U1 and is perpendicular to the line of sight LS1 of the first user U1, based on the position of the gaze point P1 of the first user U1.

[0122] According to this embodiment, the field of view FVP1 of the first user U1 is calculated based on the position of the point of gaze P1, and the degree of overlap of the field of view FVP2 of the second user U2 with the field of view FVP1 of the first user U1 is calculated. The farther the point of gaze P1 is from the first user U1 and the second user U2, the more accurately the field of view FVP1 of the first user U1 and the field of view FVP2 of the second user U2 are calculated.

[0123] Furthermore, the position of the first user U1 and the position of the second user U2 are separated from each other.

[0124] According to this embodiment, even if the users are located far apart from each other in the virtual space, the first user U1 can accurately grasp the field of view of the second user U2.

[0125] 2. Second embodiment Hereinafter, the configuration of a display control device according to the second embodiment of the present invention will be described with reference to Figs. 19 to 22. In the following description, for the sake of simplicity, the same reference numerals are used for the same components as in the first embodiment, and the description of their functions may be omitted. In addition, in the following description, for the sake of simplicity, differences between the second embodiment and the first embodiment will be mainly described.

[0126] 2.1. Configuration of the second embodiment 2.1.1. Overall Configuration Fig. 19 is a diagram showing an overall configuration of an information processing system 1A including a display control device according to a second embodiment. As shown in Fig. 19, the information processing system 1A is different from the information processing system 1 according to the first embodiment in that it includes a server 30A instead of the server 30. The server 30A is an example of a display control device.

[0127] 2.1.2. Terminal Device Configuration The terminal device 10 has a similar configuration to the terminal device 10 according to the first embodiment, and therefore a detailed description thereof will be omitted.

[0128] 2.1.3. Server configuration Fig. 20 is a block diagram showing a configuration example of the server 30A in Fig. 19. As shown in Fig. 20, the server 30A is different from the server 30 according to the first embodiment in that the server 30A includes a processing device 31A instead of the processing device 31 and a storage device 32A instead of the storage device 32.

[0129] The processing device 31A is a processor that controls the entire server 30A, and is configured, for example, using one or more chips. The processing device 31A is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 31A may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 31A executes various processes in parallel or sequentially.

[0130] The storage device 32A is a recording medium that can be read and written by the processing device 31A. The storage device 32A includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0131] The storage device 32A stores a plurality of programs including a control program PR4 to be executed by the processing device 31A, and also functions as a work area for the processing device 31A.

[0132] The communication device 33, the display 34, and the input device 35 are similar to the communication device 33, the display 34, and the input device 35 of the first embodiment, respectively, and therefore their explanation will be omitted.

[0133] The processing device 31A functions as an acquisition unit 311, a field of view image generation unit 312, a display image generation unit 313, a display control unit 314, and a reception unit 315, for example, by reading and executing a control program PR4 from the storage device 32A.

[0134] The processing device 31A differs from the processing device 31 according to the first embodiment in that it functions as a reception unit 315. The functions of the acquisition unit 311, the field of view image generation unit 312, the display image generation unit 313, and the display control unit 314 are similar to the functions of the acquisition unit 311, the field of view image generation unit 312, the display image generation unit 313, and the display control unit 314 of the processing device 31, and therefore descriptions thereof will be omitted.

[0135] The reception unit 315 receives an object in the line of sight LS1a of the first user U1, which is specified by the first user U1. For example, when the first user U1 wants to share a building in real space that the first user U1 is paying attention to with the second user U2, the first user U1 first places a pin as a virtual object at a position corresponding to the building on the display 28 of the MR glasses 20a using the terminal device 10a.

[0136] The reception unit 315 identifies the three-dimensional space coordinates of the structure in the real space based on the position of the pin placed on the display 28 by the first user U1 using the terminal device 10a.

[0137] Fig. 21 is a schematic diagram for explaining the field of view of the first user U1. As shown in Fig. 21, the first user U1 wearing the MR glasses 20a on his / her head can view the real space through the MR glasses 20a. The field of view range FV1a of the first user U1 is a quadrangular pyramid-shaped range surrounded by four boundary lines extending radially from the apex of the head, that is, the upper left end of the field of view FV1ULa, the upper right end of the field of view FV1URa, the lower right end of the field of view FV1LRa, and the lower left end of the field of view FV1LLa, with the MR glasses 20a as the apex. The three-dimensional space coordinates of the MR glasses 20a are (xu1, yu1, zu1).

[0138] The three-dimensional space coordinates of the point B1 identified based on the pin marked by the first user U1 using the terminal device 10a as the gaze point are (xb1, yb1, zb1). The line extending from the MR glasses 20a toward the point B1 is the line of sight LS1a of the first user U1. In other words, the line of sight LS1a of the first user U1 is a straight line connecting the position of the identified point B1 and the position of the first user U1. The server 30 acquires the three-dimensional space coordinates (xb1, yb1, zb1) of the identified point B1 from a position information server (not shown) connected to the communication network NET.

[0139] At point B1, a plane PL1a perpendicular to the line of sight LS1a of the first user U1 is cut into a rectangle in a quadrangular pyramid-shaped range surrounded by the upper left end FV1ULa, the upper right end FV1URa, the lower right end FV1LRa, and the lower left end FV1LLa of the field of view. In this embodiment, the rectangular cut plane PL1a is defined as the field of view FVP1a of the first user U1 based on point B1.

[0140] The shape of the field of view FVP1a of the first user U1 based on the point B1 does not have to be rectangular. Depending on the structure, shape, etc. of the MR glasses 20, the shape of the field of view FVP1a can be any shape.

[0141] 2.2. Operation of Processing Device 31A Fig. 22 is a flowchart showing the operation of the processing device 31A in Fig. 20. Hereinafter, the operation of the processing device 31A will be described with reference to Fig. 22. The routine in Fig. 22 is started, for example, when the processing device 31A is started, and is executed every time a certain period of time has elapsed.

[0142] In step S11, the processing device 31A functions as the acquisition unit 311 to determine whether or not operation information has been acquired from the terminal device 10a.

[0143] If no operation information is acquired from the terminal device 10a, that is, if the determination result in step S11 is negative, the processor 31A temporarily ends this routine.

[0144] On the other hand, when operation information is acquired from the terminal device 10a, i.e., when the judgment result in step S11 is positive, the processing device 31A functions as the acquisition unit 311, and in step S12, acquires the first visibility information from the terminal device 10a and the second visibility information from the terminal device 10b.

[0145] In step S21, the processing device 31A functions as the receiving unit 315 to receive an object in the line of sight LS1a of the first user U1, which is specified by the first user U1.

[0146] In step S22, the processing device 31A functions as the field of view image generating unit 312A to identify a plane PL1a that includes the position of the object and is perpendicular to the line of sight LS1a of the first user U1.

[0147] In step S14, the processing device 31A functions as the field of view image generating unit 312A to generate a field of view image FI0 of the second user U2 based on the first field of view information and the second field of view information.

[0148] In step S15, the processing device 31A functions as the display image generating section 313 to generate a display image DI0 based on the first visibility information.

[0149] In step S16, the processing device 31A functions as the display control unit 314 to display the display image DI0 generated by the display image generation unit 313 on the MR glasses 20a worn on the head of the first user U1, and then temporarily terminates this routine.

[0150] 2.3. Modification of the second embodiment In the above embodiment, when the first user U1 places a pin as a virtual object at a position corresponding to an object B1 such as a building, it is assumed that the object B1 such as a building is in the line of sight of the first user U1, but the object B1 does not necessarily have to be in the line of sight of the first user U1.

[0151] That is, the first user U1 may specify the object B1 in any manner. For example, the first user U1 may specify the object B1 by voice information to the processing device 31A using a microphone (not shown). In this case, the object B1 may not be present in the line of sight of the first user U1.

[0152] The reception unit 315 receives a designation of an object B1 from the first user U1 using voice information. For example, when the first user U1 wants to share the object B1, such as a building in real space, with the second user U2, the first user U1 speaks the name, address, etc. of the building.

[0153] The receiving unit 315 identifies the position of the object B1 in the real space based on the voice information uttered by the first user U1. The position of the object B1 is expressed in three-dimensional space coordinates.

[0154] The field of view image generator 312A identifies a plane PL1a that includes the position of the identified object B1 and is perpendicular to the line of sight of the first user U1.

[0155] 2.4. Advantages of the Second Embodiment According to the above description, the processing device 31A according to the second embodiment further includes the reception unit 315. The reception unit 315 receives a designation of an object B1 from the first user U1. The field of view image generation unit 312 specifies a plane PL1a that includes the position of the object B1 and is perpendicular to the line of sight of the first user U1.

[0156] According to this embodiment, when the first user U1 wants to share information about the object B1 with the second user U2, the field of view FVP2 of the second user U2 on the plane PL1a including the position of the object B1 can be easily recognized. Therefore, the first user U1 can smoothly communicate with the second user U2.

[0157] 3. Variations The present disclosure is not limited to the above-mentioned embodiment. Specific modified aspects are exemplified below. Two or more aspects selected from the following examples may be combined. In addition, the above-mentioned embodiment and the following modified aspects may be combined as long as they are not mutually contradictory.

[0158] 3.1. Variation 1 In each of the above embodiments, the field of view image FI0 of the second user U2 is an image that includes the position of the gaze point P1 on the line of sight LS1 of the first user U1 and shows the field of view of the second user U2 on a plane PL1 perpendicular to the line of sight LS1 of the first user U1. However, the field of view image FI0 of the second user U2 may be an image that includes the position (xp2, yp2, zp2) of the gaze point P2 on the line of sight LS2 of the second user U2 and shows the field of view of the second user U2 on a plane perpendicular to the line of sight LS1 of the first user U1.

[0159] Fig. 23 is a schematic diagram for explaining a field of view image FI0 of the second user U2 according to Modification 1. As shown in Fig. 23, a plane PL3 is a field of view FVP3 of the second user U2 on a plane that includes the position (xp2, yp2, zp2) of the gaze point P2 and is perpendicular to the line of sight LS1 of the first user U1.

[0160] The plane PL1 and the plane PL3 are parallel to each other, so they do not intersect. The overlapping portion OL2 is defined as the portion where the field of view FVP1 of the first user U1 and the field of view LS3 of the second user U2 overlap when viewed from a direction parallel to the line of sight LS1 of the first user U1.

[0161] In this way, the second view information includes the position of the gaze point P2 of the second user U2. The view image generation unit 312 specifies a plane PL3 that includes the position of the gaze point P2 of the second user U2 and is perpendicular to the line of sight LS1 of the first user U1, based on the second view information.

[0162] According to this embodiment, the overlapping part OL2 with the field of view FVP1 of the first user U1 can be obtained even by using a pseudo field of view that passes through the position of the gaze point P2 of the second user U2 and is perpendicular to the line of sight LS1 of the first user U1.

[0163] 3.2. Variation 2 In each of the above embodiments, each user operates the MR glasses 20 using the terminal device 10 owned by each user, but if various operations on the MR glasses 20 can be performed using the MR glasses 20, the terminal device 10 is not necessarily required. Also, if the terminal device 10 is not used, the MR glasses 20 may be directly connected to the communication network NET.

[0164] 3.3. Variation 3 In the first embodiment, the radius ru1, which is the distance from the MR glasses 20a to the gaze point P1, is a predetermined distance, but the radius ru1 may be the distance to the object closest to the first user U1 among the objects intersecting the line of sight LS1 of the first user U1. The position information of the object closest to the first user U1 may be acquired from a position information server (not shown).

[0165] 3.4. Variation 4 In each of the above embodiments, the information processing systems 1 and 1A have two users, but the number of users may be three or more.

[0166] 4.Other (1) In the above-described embodiment, storage device 12, storage device 22, storage device 32, and storage device 32A are exemplified by ROM and RAM, but the storage devices may be flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media. The programs may also be transmitted from a network via electric communication lines. The programs may also be transmitted from a communication network NET via electric communication lines.

[0167] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0168] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

[0169] (4) In the above-described embodiments, the determination may be made based on a value represented using one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0170] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0171] (6) Each function illustrated in Fig. 1 to Fig. 23 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0172] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0173] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0174] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.

[0175] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0176] (10) In the above-described embodiment, the server 30 and the server 30A may be a mobile station (MS). A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In the present disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

[0177] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read with "access". As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0178] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0179] (13) The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0180] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.

[0181] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.

[0182] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0183] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to explicit notification, but may be implicit (e.g., not notifying the specific information).

[0184] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is for illustrative purposes only and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0185] 1, 1A...information processing system, 10, 10a, 10b...terminal device, 20, 20a, 20b...MR glasses, 30, 30A...server, 312...view image generation unit, 313...display image generation unit, 314...display control unit, 315...reception unit, DI0...display image, FI0...view image, FVP1, FVP1a...first user's view, FVP2...second user's view, LS1, LS1a...first user's line of sight, LS2...second user's line of sight, PL1, PL1a, PL2, PL3...plane, U1...first user, U2...second user

Claims

1. a field of view image generating unit that generates a field of view image showing a field of view of the second user in a plane perpendicular to the line of sight of the first user based on first field of view information related to the field of view of the first user including the line of sight of the first user and second field of view information related to the field of view of the second user; A display image generating unit that generates a display image in which the field of view image is arranged in the field of view of the first user based on the first field of view information; A display control unit that displays the display image on a glass device worn on the head of the first user; A display control device comprising:

2. The first visibility information includes a position of a gaze point of the first user, The display control device according to claim 1 , wherein the field of view image generation unit identifies a plane that includes the position of the first user's point of gaze and is perpendicular to the first user's line of sight based on the position of the first user's point of gaze.

3. a reception unit that receives designation of an object from the first user, The field of view image generator identifies the plane including the position of the object. The display control device according to claim 1 .

4. The second field of view information includes a position of a gaze point of the second user, The field of view image generation unit specifies a plane including a position of the gaze point of the second user and perpendicular to the line of sight of the first user based on the second field of view information. The display control device according to claim 1 .

5. The location of the first user and the location of the second user are separated from each other. The display control device according to claim 1 .

Images